1.  April 01 2025 Price Increase

Starting April 1, 2025, new and renewing monthly billed annual contract subscriptions will cost 5% more than annual upfront payment subscriptions.

2. Changes to billing frequency and terms

•  Mid-term billing frequency changes for triennial offers will no longer be allowed
• Mid-term billing frequency changes from monthly billing on annual subscription term can not be changed to yearly. The billing change to yearly can only be made prior to renewal subscription expiry term date.

3. MFA requirement for Microsoft 365 admin centre

On February 3, 2025, Microsoft is going to start enforcing MFA for all users who access the Microsoft 365 admin center portal. Review the latest information and timeline.

4. Office 2016/2019 EOS in October 2025

On October 14, 2025, Microsoft support will end for all Office 2016 and 2019 suites, standalone apps, and servers.  After that date, Microsoft will no longer provide security fixes, bug fixes, or technical support for these products.  See the product list here.

As part of Microsoft’s pricing policy, Microsoft increased its prices on September 1, 2023.
The increase, impacted Australian, New Zealand, Canadian, and Swiss Microsoft customers. This price increase has been put in place to ensure greater pricing consistency globally in line with prices in US dollars.

For Australia, the costs associated with Microsoft solutions in the cloud and on-premises, has now increased by 9%.
Here’s the official Microsoft announcement for your information.

Conetix as of 01 March 2024 will be bringing all Microsoft services in line with these pricing changes.

1.  March 01 2024 Price Increase

For both Microsoft direct and all resellers, Microsoft has already increased their subscription prices for Microsoft 365 on 01 September 2023.

All other product prices will also be subject to price rise of 9%

2. Contract changes from Microsoft

To date, Conetix has provided flexible pricing with month-by-month agreements for all Microsoft licensing at the annual discounted rate. For example, if you purchase from Microsoft direct the prices shown are with an annual commitment and without GST:

While we understand none of this is of benefit to consumers, unfortunately it’s a worldwide Microsoft change so all users are affected.

We’ll be doing our best to make it as smooth as possible so that your focus can be back on your business and not billing changes.

As you may be aware, OzeWebHelp is partnering with Conetix to offer to provide your Web Hosting Services going forward and we are excited to help the team at OzeWebHelp as they move into their next phase of business with Made by Hudson.

We know you loved the service and support you received from OzeWebHelp over the past years and like OzeWebHelp we are committed to providing you with not only great value for your web hosting but exceptional customer service.

With over 20 years of experience in the web hosting industry Conetix has built our business on our dedicated support, intelligent automation systems and superior hardware to provide our clients with an uncompromised web hosting experience.

Conetix Pty Ltd is a 100% Australian owned and operated company, focused on providing enterprise grade solutions ranging from basic web hosting through to advanced application and desktop delivery.

Our servers are located at the state-of-the-art facility iSeek in Brisbane and our friendly support team is Queensland based, operating from the city of Ipswich.

So what happens next?

We will plan to migrate all services in stages over the coming months. We will aim to migrate your OzeWebHelp services over to the Conetix Cloud platform from 01 April 2022 and onward and plan to complete all migrations by 30 June 2022.

We understand changing hosting providers can be stressful. As such your existing payments and service plan costs to OzeWebHelp are being honoured by Conetix and the Conetix staff are here to assist fully with all the work with the migration and any of your hosting questions.

How does the migration work?

Once your migration has been scheduled we will notify you via email and one of our dedicated migrations team members will migrate your services over to the Conetix Cloud Platform.

We’ve built iron clad processes around migrating websites, email, DNS and domain names. Our migrations team will keep you up-to-date with the progress and in most instances the migrations are completed with zero disruption. If an issue is hit where some disruption can’t be avoided (for example, we may need to pause eCommerce transactions for up to 10 minutes), the team will consult with you to come to a mutually agreed time to complete this.

Some Key Points

Both OzeWebHelp and Conetix are commited to providing you with a stress free migration and ongoing hosting experience.

• No Downtime Migration action by Conetix’s dedicated migration team
• Conetix will honour any prepaid accounts
• Conetix will honour your existing OzeWebHelp plans and pricing

Frequently Asked Questions

Once again, Welcome to Conetix and please feel free to call or email if you need additional information relating to your hosting or other web services we provide.

Photo by Katherine Hood

There are two critical changes to your subscription.

1.  March 2022 Price Increase

For both Microsoft direct and all resellers, Microsoft is increasing their subscription prices for Microsoft 365 on 01 March 2022. These prices have already been announced for the US market, however they haven’t been finalised for the Australian market yet. At present, the US price increases have been:

All other product prices will remain the same at this stage. As soon as we have final pricing from Microsoft, we’ll ensure we let everyone know.

2. Contract changes from Microsoft

To date, Conetix has provided flexible pricing with month-by-month agreements for all Microsoft licensing at the annual discounted rate. For example, if you purchase from Microsoft direct the prices shown are with an annual commitment and without GST:

As of 01 March 2021, to maintain the annual discount you must have a contract for your subscription. This contract has a minimum 12 month term and will be per subscription (e.g. per email account). Conetix is currently modifying our billing process to ensure this is as easy as possible.

Alternatively, if you require month-by-month flexibility (so that you can cancel at any time) for your subscription then there will be a 20% monthly premium. This is the same rate if you buy from Microsoft directly or from any other reseller and unfortunately we don’t have any way around this.

What do I need to do now?

Once details have been finalised by Microsoft and their distributors, we’ll be in contact again to confirm prices and the next steps. For now, this email is simply intended to inform you of the changes and to prepare to make changes within the next two months.

We’re continuing to work with our Australian distributor and will ensure we provide further communication as soon as we have any further information.

While we understand none of this is of benefit to consumers, unfortunately it’s a worldwide Microsoft change so all users are affected.

We’ll be doing our best to make it as smooth as possible so that your focus can be back on your business and not billing changes.

Urgent Update

If you have received an email and call that your nameservers need to be updated but have not done so, this may be the cause of any website and email outages.

If you have been instructed to do so, please ensure namservers have been updated to the Conetix systems (ns1.conetix.com and ns2.conetix.com) to ensure continuity of service.

Welcome!

As you may be aware, StudioCoast is partnering with Conetix to offer to provide your Web Hosting Services going forward.

We know you loved the service and support you received from StudioCoast over the years and like StudioCoast we are committed to providing you with not only great value for your web hosting but exceptional customer service.

With over 20 years of experience in the web hosting industry Conetix has built our business on our dedicated support, intelligent automation systems and superior hardware to provide our clients with an uncompromised web hosting experience.

Conetix Pty Ltd is a 100% Australian owned and operated company, focused on providing enterprise grade solutions ranging from basic web hosting through to advanced application and desktop delivery.

Our servers are located at the state-of-the-art facility iSeek in Brisbane and our friendly support team is Queensland based, operating from Ipswich City.

So what happens next?

Once you have opted in to have your services migrated to Conetix, we will plan to migrate all services in stages over the coming months. We will aim to migrate your StudioCoast services over to the Conetix Cloud platform from January 2021 and onwards into the new year.

We understand changing hosting providers can be stressful. As such your existing payments and service plan costs to StudioCoast are being honoured by Conetix and the Conetix staff are here to assist fully with all the work with the migration and any of your hosting questions.

How does the migration work?

Once your migration has been scheduled we will notify you via email and one of our dedicated migrations team members will migrate your services over to the Conetix Cloud Platform.

We’ve built iron clad processes around migrating websites, email, DNS and domain names. We do not expect any interruption to services, though if an interruption is required our migration team will communicate this to you.

Frequently Asked Questions

Q: Do you support Windows Hosting?
A: YES. We support both Windows Shared and your own Virtual Private Server (VPS), including an easy to use management interface.

Q: Will Conetix do the migration for me?
A: YES. We have a dedicated migration team that will do all the heavy lifting and move your sites for you and keep you informed every step of the way. It is also at no cost to you.

Q: When will my migration occur?
A: Our migrations team are currently working through all of the migrations as quickly as possible. While we’d love to provide you with a detailed schedule, due to the variability of each website there’s no easy estimation we can provide. However, we are prioritising some migrations or scheduling for exact times where we can make it as easy as possible for your business. Please just contact our support team to coordinate a schedule to help minimise disruption.

Q: Will my migration be completed in time?
A: Yes. While the initial date for all migrations was set as 30 May 2021, this has been extended until 30 June 2021 to ensure all customers who have opted in are across into our platform and working without issue.

Q: Are you keeping my StudioCoast plan?
A: Yes. We are honouring all existing StudioCoast plans for as long as you like. Any upgrades or changes to the plans will be at the Conetix rates however we always confirm pricing with you first. We’re happy to help optimise your billing as well and one of our support team members can review your account an usage to work out the best deal for you.

Q: I have prepaid my account with StudioCoast, what happened there?
A: Conetix is honouring any pre payments made to StudioCoast.

Q: Will there be downtime?
A: Most migrations can be completed without any downtime or disruption. If downtime is required, you’ll be informed of ahead of time and this will be scheduled with you.

Q: Once I OptIn, do I need to cancel my account with StudioCoast?
A: No, we will cancel your account once fully migrated.

Q: Do you support Umbraco CMS? 
A: Yes. We support Umbraco on both our shared hosting environment as well as on a dedicated VPS.

Q: We you be transferring all existing SSL certificates from Studio Coast to Conetix?
A: Yes and we also offer a Free Let’s Encrypt SSL certificate with all hosting on our platform.

Once again, Welcome to Conetix and please feel free to call or email if you need additional information relating to your hosting or other web services we provide.

We will continue to monitor and adjust our approach daily as any new information or government directives come through and will update this blog if any details change.

Update: As of Monday 23 March 2020, Conetix has enacted their work from home policy to ensure staff remain infection free and help avoid any accidental transmission. As outlined below, our team remains 100% available and operational.

Update #2: As of 06 July 2020, the Conetix team returned to the office with two weeks clear of community transmissions within the state. We have the current luxury of a spacious office which allows greater than 1.5m between staff members and have sanitation stations installed at all entrances and exist to help keep our staff safe.

Will this cause any issues with my website, server or email?

No. From a server perspective, we always build in redundancy to our platforms so that any one failure doesn’t result in an outage. In the event that a server does require repair, spares for the servers are held within a separate warehouse within 1km from the data centre. Again, this repair would be conducted without any service disruption.

We also have sufficient network capacity that even if every website and email had triple the network usage it does now, there would be no slowdowns at all. Like always, Conetix continues to monitor this closely to ensure there aren’t any issues.

Will support still be available?

Yes. Conetix already has contingency planning in place so that if staff need to be isolated or government orders direct us not to travel to work, we can operate normally. Staff members are able to work from home or other environments securely and without any disruption to the service we provide. Conetix has been updating and refining our security policies to help facilitate a fault free transition to any 100% emergency remote work scenarios.

Our support number will remain the same (1300 789 260) and staff will still be able to take calls. Please just use our contact form or email (support@conetix.com.au) to send written requests through.

As is the case with any general issues, Conetix reports these via our online status page at conetix.online and if there are any unforeseen issues they will be reported here as well.

Update: 23 March 2020: Some disruption to incoming calls exist due to capacity issues with our phone provider. Please see the status page at conetix.online for more information. 100% of our team is still able to return calls quickly if you email us to request a callback.

Do I need to do anything to prepare?

We would recommend that if you work in a centralised office, you also consider the scenario of having to work from home or work from a different PC. If you wish to check your email from a different computer to normal, you’ll need a copy of your email password to set it up.

Webmail access

There’s two easy ways to access your email remotely if you have your services with Conetix:

You may also need to consider other providers (outside of Conetix) who provide critical services to your business as they may be impacted as well. It would be a good idea to produce a list of companies who you do work with and just double check their contingency plans as well.

For all medical related questions, we recommend health.gov.au’s COVID-19 page: https://www.health.gov.au/news/health-alerts/novel-coronavirus-2019-ncov-health-alert

Need help? Advice?

Conetix has always been on the front foot when it comes to technology and we’re happy to help during this time to provide extra advice on what other steps you can take in your business.

Please just give us a call on 1300 789 260 if you have any questions or if there’s any way we can assist.

Photo by CDC on Unsplash

Let’s Encrypt is a free, automated, and open Certificate Authority (CA), run for the public’s benefit. Run by the Internet Security Research Group (ISRG), Let’s Encrypt is made possible by major backing from sponsors like the Mozilla Foundation, Akami, Chrome, Sucuri, Cisco Systems and other sponsors.

One of the key principles behind Let’s Encrypt is that it is free. It allows for every person or business who owns a domain name to secure their website using a simple, automated install of a SSL certificate. More importantly, the process of applying for and validating the certificate is through an automated system therefore removing the clunky email based validation many other CA’s still use.

Since Let’s Encrypt BETA began in September 2015 they have issued more than 4.2 million certificates. Let’s Encrypt left BETA on April 26, 2016 and continues to grow at a rapid pace.

Source: Let’s Encrypt

As a Plesk partner we were able to test and use Let’s Encrypt before it was in general release. We were immediately impressed on how simple and easy it was to issue a certificate via the Let’s Encrypt Plesk button. We could see immediate benefits to our customers who can now secure their website in three easy clicks.

Some of the key benefits of the Let’s Encrypt certificates :

• Zero Cost SSL
• Easy installation using Plesk
• Instant Validation
• Automated renewal process
• No dedicated IP required (which is extra money)
• Trusted by all major browsers

Why is it important?

There are a number of important factors why you should use an SSL certificate for your website, the obvious one is security and trust. When someone visits your website instead of seeing http in their browser they get presented with https:// and the green security lock.
The transmission of information from your website to the clients browser and vice versa is then securely transmitted.

One other important factor is Google now uses HTTPS as a ranking signal. When every bit counts in your website rankings, improving your security and rankings for your website couldn’t be easier with Let’s Encrypt.

If you are a current Conetix client and wish to take advantage of Let’s Encrypt free SSL you can login to our control panel and simply click on Let’s Encrypt Icon:

This will then ask for your email address (to send expiry notices to) and immediately validate and generate the certificate for you.

Success!!

If you are on one of our Virtual Private Servers or Containers running Plesk you can install Let’s Encrypt from Plesk Extensions – Here is a link to our support article on How to install Let’s Encrypt Extension on Plesk 12.5

For those that wish to use a domain validated SSL certificate we still have these options available including wildcard and full EV certificates.

Feel free to contact anyone of our team members for advice on which SSL is best for your business.

Established in 2000, Conetix is an Australian service provider that specializes in delivering enterprise-grade solutions ranging from web hosting to advanced application and desktop delivery. This is backed by dedicated local support and a strong focus on service quality. Conetix also provides web developers and Internet service providers with wholesale services and builds solutions tailored to their business needs. 

Business Challenge 

As its business expands, Conetix has been careful to ensure that this does not come at the expense of customer service. “Our business ethos and value is to deliver high-quality service,” said Jamin Andrews, Managing Director at Conetix. “When the phone rings, someone will answer it promptly. When there is an enquiry or if there is a technical support issue, it will be resolved very quickly. We pride ourselves in our service.” 

The ability to maintain customer service levels while scaling its operations involves both platform and personnel – having the right platform to support its growing customer base, and personnel who are well-equipped with both the technical expertise and customer-centric mindset to ensure a high standard of service delivery. “We want to make sure that we grow in both these areas, across the board,” added Mr. Andrews. 

The Solution 

To achieve this, Conetix deployed Parallels Cloud Server (PCS) in April 2013 to consolidate its IT environment and scale its operations. 

Conetix’s confidence in PCS stemmed from its long partnership with Parallels, dating back to the early days of its operations. At that time, Conetix was looking at a number of tools that could give it a competitive advantage in the marketplace. It adopted Parallels Plesk Panel and later went on to deploy other products from the Parallels portfolio, such as the billing and hosting automation solution Parallels Business Automation (PBA). In fact, Conetix was one of the first service providers in Australia to run PBA as well as the operating system virtualization solution Parallels Virtuozzo Containers. 

Over time, Conetix also deployed virtualization products from other vendors. However, Conetix soon realized that this was “not a cheap exercise”. Training its staff on the new products was costing Conetix thousands of dollars a year. There were also issues with under-utilization of resources, as these had to be managed outside of the PBA management and provisioning platform. 

With the latest PCS deployment, Conetix has come back to the fold with Parallels, said Mr. Andrews. 

PCS is a virtualization solution that allows both containers and hypervisors to be deployed on the same physical server. This allows Conetix to take advantage of the scalability and streamlined operations of operating system virtualization (containers), while also making use of high-performance virtual machines (hypervisors) for isolated applications or simultaneous heterogeneous environments. 

The deployment was swift and hassle-free. “The return on investment, from deploying the server to having customers on it, was virtually immediate,” said Mr. Andrews. 

Benefits 

“Parallels Cloud Server does all the work for us,” said Mr. Andrews. “It is a platform that everyone knows and trusts. From a scalability point of view, we are certain that it is the best platform to continue forward with as it allows us to focus our energy on our customers and on the company’s growth instead of worrying about technical issues.” 

Enhanced Scalability 

With PCS, Conetix has been able to keep pace with business growth, deploying 10 new physical nodes within the space of two months. It is also able to provision a new Hypervisor virtual server for a customer within minutes. 

Centralized Management and Service Delivery 

The consolidation of Conetix’s IT environment on PCS and its integration with PBA has also paved the way for a seamless customer experience with management, provisioning and other services centralized in one location. 

From an operational perspective, the consolidation reduced the technical effort required to manage and maintain disparate systems. “Rather than have multiple systems to maintain, we can now manage it all using one core platform,” said Mr. Andrews. 

Reduced Maintenance Windows 

PCS features rebootless kernel updates, which means that Conetix customers experience much shorter maintenance windows and hence higher availability on their systems. Instead of the half-hour that it used to take to run updates for its servers, PCS allows the task to be completed within minutes. 

More Efficient Use of Resources 

When a new server is added to the PCS platform, it automatically becomes part of the Conetix cloud pool and its utilization is optimized through the deployment of Hypervisor instances as well as containers. In comparison, if a new server were to be deployed on another platform, the server could be half-utilized because Conetix would have to grow into it. 

Integrated Backup 

Built-in backup capabilities in PCS eliminate the need for Conetix to invest in third-party products to deliver the same level of backup services. 

With computation decoupled from storage, containers and virtual machines can be instantly migrated to an operational physical server if the original physical server becomes unavailable. Similarly, if a hard drive fails, containers and virtual machines on the effected server can continue to operate using data replicated on other hard drives in the storage cluster. 

This feature translates into significant cost savings compared with the use of other virtualization products that require additional licenses for backup solutions. 

Massive Reduction in Training Costs 

The adoption of PCS has resulted in “massive” savings for Conetix in terms of manpower training costs. Previously, with the adoption of virtualization products from other vendors, the technical team had to be trained on new platforms. This could work out to a few thousand dollars a year for each technician/engineer. This cost was virtually eliminated with PCS, which allows the IT environment to be managed using the same set of Parallels tools that the technical team was already familiar with. 

For full Case Study you can download HERE.

Established in 2000, Freom Pty Ltd is a web design and hosting company with customers ranging from small and medium enterprises to large multinational corporations in Australia. Through its hosting packages, it provides businesses with the storage and email accounts needed to create a professional Internet presence. For customers with more unique requirements, Freom also provides them with the flexibility to create their own packages.

Business Challenge                   

As a web hosting company, Freom places great importance on the security of its customers’ web assets. In particular, it wanted to make sure that its clients’ systems were protected from spam mail and other online threats.                  

Freom also faced the challenge of managing different products within its hosting environment. For example, its customers were using various versions and

brands of control panels, which required different skillsets and significant effort to configure and maintain. Furthermore, many of its customers were beginning to use WordPress to manage their website content, so Freom had to find a way to help them manage WordPress more efficiently and securely as well.

Solution                  

To address these issues, Conetix introduced Freom to their new Plesk 12 solution offer designed for web agencies. Conetix is an Australian service provider focused on delivering enterprise grade solutions ranging from web hosting to advanced application and desktop delivery. It has also been a Parallels partner for over                   

10 years. As a Conetix reseller, Freom was able to tap into Conetix’s expertise in Parallels products and gain better insights into Parallels’ capabilities.                    

The Solution for Web Professionals includes a powerful set of website management tools, which includes an enhanced security core that allows service providers and

web agencies to secure their infrastructure while reducing support costs. It integrates web application firewall capabilities with a service that scans log files and bans IP addresses that exhibit malicious behaviour. These are combined with outbound antispam and server hardening tools to deliver server-to-site protection out-of-the-box.

In addition to these, Parallels Plesk includes a next-generation content distribution network (CDN), powered by CloudFlare, which brings content closer and faster to web site visitors. The Web Pro and Web Host editions of the new Parallels Plesk 12 also come with a WordPress Toolkit that simplifies the daily tasks required to manage and secure WordPress sites.  After evaluating Contetix solution offer with Parallels Plesk and comparing its capabilities with other offerings in the market, Freom decided to standardize on the use of Parallels Plesk for all of their web hosting clients.             

“Since partnering with Conetix doing business has become a much more seamless process. We are now able to effortlessly integrate all our clients into the one offering resulting in consistency across the board.”  

— Rick Jewson, Owner, Freom Pty Ltd 

Benefits                   

With the latest version of Parallels Plesk 12, Freom has been able to address the security and management issues that concern many web hosters and deliver these capabilities to its customers as well. Through its partnership with Conetix, it has also been able to leverage the latter’s deployment of Parallels’ infrastructure and automation solutions to deliver greater reliability, high service availability and more proactive customer support.                            

Peace of mind with enhanced security                    

Parallels Plesk gives Freom and its customers “real peace of mind” by resolving spam mail and other security issues. With the new Plesk security core, outgoing spam protection ensures cleaner IP addresses and prevents its customers’ servers from being blacklisted. The security core also protects Freom and its customers against persistent attacks targeting known or newly-discovered vulnerabilities, and helps increase uptime as malicious attacks against servers are automatically blocked in real time.                    

Simplified system management                   

With the new Parallels Plesk, Freom’s customers now get a simplified view of their systems, enabling them to manage their web environment without the need for technical skills. It also gives them the flexibility to configure and manage many of the regular features that they need. For example, they are able to set up email accounts for their employees without having to turn to their service provider. This, in turn, frees up Freom to focus its attention on more critical customer issues.                    

Simplified WordPress management                   

By simplifying the daily tasks required to manage and secure WordPress sites, Parallels Plesk also reduces the time and effort required for system administration and enables Freom’s customers to focus on their core business. The WordPress Toolkit provides web site owners with a single point of entry to manage multiple WordPress installations, plugins and themes and enables them to secure the system easily by applying the most common recommended security settings.                    

Improved performance

The integration of a next-generation CDN also enables Parallels Plesk to deliver faster site performance and bandwidth savings. The CDN brings content closer and faster to visitors, enabling web sites to load twice as fast on average and saving end users up to 60% in bandwidth, without requiring any configuration or additional setup.                    

High availability and reliability                  

Backed by Conetix’s implementation of Parallels Cloud Server and Parallels Cloud Storage, Freom is also able to provide its customers with a highly-reliable infrastructure. The Parallels cloud platform enables applications to be migrated instantly to a working physical server if the original physical server becomes unavailable. In the same way, if the hard drive in a server fails, the applications on that server can continue to operate using data replicated on other hard drives in the storage cluster, thus ensuring high service availability.                    

Proactive support                    

Proactive support from Conetix has played an important role in helping Freom to reach out successfully to new customers. For example, using Parallels Business Automation, Conetix is able to quickly spin up new virtual private servers which Freom can then offer to its clients for a free trial, boosting the opportunity to close a deal.

You can download the case study in PDF form here

In this article, I’ll be covering some of the details of the metric and log collection. As outlined in the first guide, we use a combination of:

• PRTG
• Graphite
• Graphana
• RabbitMQ
• Elasticsearch
• Logstash
• Kibana
• Panopta

General Monitoring

While it may seem like a large list of tools and programs, each covers some very specific use cases. We use PRTG to provide our overall system health, with nearly 1000 sensors monitoring all of our systems. This is a very comprehensive system which can cover deep down into the application layer as well as being very reliable. In our case, we use it to monitor all of our main hardware nodes, dedicated servers for clients and networking infrastructure. For our VPS’s we have our own system for metrics and monitoring, which is detailed further on in this article.

We then use Panopta for external monitoring, so that we have independent data as well as regional specific checks so that we can rule out false positives. Australia’s connectivity to the rest of the world can be choked at times, which results in packet loss. Other alerting systems don’t have a Point of Presence (POP) in Australia, which would result in false positive “outage” alerts. 99% of our market is aimed at and located in Australia, so although a brief international connectivity issue isn’t great, it’s also non-critical.  Thankfully, Panopta have 4 POP’s in Australia which means we don’t get woken at 2am anymore for false positives!

As both systems are essentially “out of the box”, there’s not a lot that we customise. PRTG we simply setup custom maps (not the geographic type) to display on our NOC screens but otherwise it simply just runs.

Detailed Logging

For anyone who’s job involves reading logs or analysing logs, the ELK stack (Elasticsearch / Logstash / Kibana) is one set of tools which you’ll want to give a go. Here’s a quick overview of the components:

• Elasticsearch is essentially your logging and search system, which uses JSON data for storage. This is perfect for logs and the search speed is very impressive.
• Logstash does the collection and has plenty of options for reading files as well as many other systems and then has a variety of options to process the data and forward to external systems (like Elasticsearch).
• Kibana is a pure HTML / JS web interface which talks to your Elasticsearch server. This is where you can configure displays with a number of different graphs, widgets and log views.

Since the installation is covered well here, I won’t go into detail within this blog. As long as you have a working Java installation, the installation is quite trivial.  Just be sure to allow plenty of storage, memory and processing if you want to index large volumes of logs or at least have the capacity to increase these resources.

We also use Curator to clean up the indices based on set time frames. As logging a lot of data can result in hundreds of gigabytes of data, this is a crucial one to implement. Depending on the type of log, you can set different retention periods or simply set a global one. For example, if you only wanted to hold the indices for 30 days, you could use Curator like this:

curator delete --older-than 30

This will simply clear all indices older than 30 days. There are other options available, make sure you check the documentation.

Fortigate Logging

One of the first items we wanted to be able to see detail logging data on was the Intrusion Prevention System (IPS) for our Fortigate firewalls. While Fortigate have a basic dashboard in which to view the data, we wanted to take it much further and generate custom views based on our exact requirements.

With events like Heartbleed and ShellShock, we were able to create specific dashboards to filter and view only this data related to that event. This way, we were able see the size of the threat as well as ensuring the firewalls were blocking it.

The first step is to collect the logs. The easiest way is to have the Fortigate units forward their data via the standard Syslog format. Fortigate have a handy guide on their log format here, and you’ll certainly need it! Our logging has progressed from the 5.0 format through to the 5.2 log format and while Fortinet claim they’ve standardised the logging more with the 5.2 release, it’s still very fiddly. The examples below will be for 5.2, however there are links to gists for the 5.0 format as well.

Here’s how our system is currently running:

The first step was to configure the Fortigates. Here’s the Syslog format:

config log syslogd setting
    set status enable
    set server "192.168.xxx.xxx"
    set port 514
end
config log syslogd filter
    set severity error
end

One thing to note is that the log level is set to error. The reason for this is that by default, the Fortigate systems will log all sessions via syslog and this will result in a significant amount of data. Storing session data in Elasticsearch was generating hundreds of gigabytes a week and taking a considerable amount of resources to do so. For now, we’re simply using the NetFlow data through other software to conduct IP accounting and detailed network analysis.

On the Logstash server, there are two elements to the configuration. The first is the overall config:

filter {
  if [type] == "syslog" {
    grok {
     patterns_dir => ["/etc/logstash/patterns/"]
      match => ["message" , "%{FORTIGATE_52BASE} %{FORTIGATE_52IPS}"]
      add_tag => ["fortigate"] 
    }
    grok {
       patterns_dir => ["/etc/logstash/patterns/"]
       match => ["message" , "%{FORTIGATE_52BASEV2} %{FORTIGATE_52DOS}"]
    }
  } 

output {
  elasticsearch_http {
    host => "IP of the Elasticsearch Server"
  }
}

Essentially, this is a fairly vanilla Logstash file. Most of the work is done in the log patterns. This is in a grok format, which takes a bit of getting used to but if you’ve worked with regular expressions before then this is very close. These patterns turn unstructured data into structured data so that it can be easily logged and therefore queried.

I found a few other references where others had used Logstash to log the Fortigate data, however none of them worked perfectly for me. Rather than stopping and starting Logstash to debug getting the patterns to work, there’s a handy Grok Debugger online, which is immensely helpful when first getting started.

With a few hours of fiddling, I managed to get a working solution:

FORTIDATE %{YEAR:year}-%{MONTHNUM:month}-%{MONTHDAY:day}

FORTIGATE_52BASE <%{NUMBER:syslog_index}>date=%{FORTIDATE:date} time=%{TIME:time} devname=%{HOST:hostname} devid=%{HOST:devid} logid=%{NUMBER:logid} type=%{WORD:type} subtype=%{WORD:subtype} eventtype=%{WORD:eventtype} level=%{WORD:level} vd="%{WORD:vdom}"

FORTIGATE_52BASEV2 <%{NUMBER:syslog_index}>date=%{FORTIDATE:date} time=%{TIME:time} devname=%{HOST:hostname} devid=%{HOST:devid} logid=%{NUMBER:logid} type=%{WORD:type} subtype=%{WORD:subtype} level=%{WORD:level} vd="%{WORD:vdom}"

FORTIGATE_52IPS severity=%{WORD:severity} srcip=%{IP:srcip} dstip=%{IP:dstip} sessionid=%{NUMBER:sessionid} action=%{DATA:action} proto=%{NUMBER:proto} service=%{DATA:service} attack="%{DATA:attack}" srcport=%{NUMBER:srcport} dstport=%{NUMBER:dstport} attackid=%{NUMBER:attackid} profile="%{DATA:profile}" ref="%{DATA:ref}";? incidentserialno=%{NUMBER:incidentserialno} msg="%{GREEDYDATA:msg}"

FORTIGATE_52DOS severity=%{WORD:severity} srcip=%{IP:srcip} dstip=%{IP:dstip} sessionid=%{NUMBER:sessionid} action=%{DATA:action} proto=%{NUMBER:proto} service=%{DATA:service} srcintf="%{HOST:srcintf}" count=%{NUMBER:count} attack="%{DATA:attack}" srcport=%{NUMBER:srcport} dstport=%{NUMBER:dstport} attackid=%{NUMBER:attackid} profile="%{DATA:profile}" ref="%{DATA:ref}";? msg="%{GREEDYDATA:msg}" crscore=%{NUMBER:crscore} craction=%{NUMBER:craction}

You can quickly get a copy of the patterns , I have the FortiOS 5.2 Gist: here  and FortiOS 5.0 Gist here.

Update: After yet another log format change even with an incremental update to 5.2.3, we’ve swapped over to using the kv filter within Logstash. Here’s what we’re using: 

kv {
    add_tag => ["fortigate"]
}

Many of the problems came from the fact that the log format seems to change just enough to be annoying. Some items are in quotes whereas others aren’t, so this took a further few hours to sort out. Hopefully, anyone who hasn’t tried it yet can benefit from all the fiddling I’ve already had to do!

If I spend a bit more time I could probably combine a few more of the patterns into one, however once I had it working perfectly I didn’t want to touch it.

Apache Logging

We have hundreds of Apache based webservers within our infrastructure and we’ve enabled centralised logging for a few systems so far. Initially, we’re just logging our own systems to help determine any odd patterns but have plans to push this out to all of our systems.

Since the Apache log file format is very well defined and known, Logstash has the grok pattern as a default inclusion. To log the data, we use the following:
 

if [type] == "apache" {
    grok {
      type => "apache"
      pattern => "%{COMBINEDAPACHELOG}"
    }
    mutate {
      add_field => { "clientreverse" => "%{clientip}" }
    }
    
    dns {
      reverse => ["clientreverse"]
      action => ["replace"]
    }

    date {
      type => "apache"
      # Try to pull the timestamp from the 'timestamp' field (parsed above with
      # grok). The apache time format looks like: "18/Aug/2011:05:44:34 -0700"
      match => [ "timestamp", "dd/MMM/yyyy:HH:mm:ss Z" ]
    }
}

While the basic grok matching is very simple, we then do two further manipulations. The first is to add a field via mutate and perform a reverse dns on it. Rather than just seeing the IP, we can now view the reverse DNS as well as the ability to search via it. When looking at traffic anomalies, this becomes quite useful for quickly visualising normal traffic from foreign attacks. We can also use Logstash to do a geographic lookup of the IP to give us per country / city based data and we’ll be implementing this over the next few weeks.

Postfix Logging

Another area we’re collecting data on is logging Postfix data. This is the standard MTA we use on most of our platforms and when you start reading the logs it’s a bit of a mess. For starters, it uses multiple lines for the same event, so you have to be able to handle parsing when all of the data isn’t initially available. The good news is that Logstash already handles some of these events, albeit in a slightly convoluted way.

This is our Logstash config for Postfix:

input {
  file {
    type => "postfix"
    path => "/var/log/maillog"
  }
}

filter {
    grok {
        type => "postfix"
        patterns_dir => [ "/etc/logstash/patterns" ]
        pattern => [
            "%{SYSLOGBASE} %{POSTFIXSMTPDCONNECTS}",
            "%{SYSLOGBASE} %{POSTFIXSMTPDACTIONS}",
            "%{SYSLOGBASE} %{POSTFIXSMTPDTIMEOUTS}",
            "%{SYSLOGBASE} %{POSTFIXSMTPDLOGIN}",
            "%{SYSLOGBASE} %{POSTFIXSMTPDCLIENT}",
            "%{SYSLOGBASE} %{POSTFIXSMTPRELAY}",
            "%{SYSLOGBASE} %{POSTFIXSMTPCONNECT}",
            "%{SYSLOGBASE} %{POSTFIXSMTP4XX}",
            "%{SYSLOGBASE} %{POSTFIXSMTP5XX}",
            "%{SYSLOGBASE} %{POSTFIXSMTPREFUSAL}",
            "%{SYSLOGBASE} %{POSTFIXSMTPLOSTCONNECTION}",
            "%{SYSLOGBASE} %{POSTFIXSMTPTIMEOUT}",
            "%{SYSLOGBASE} %{POSTFIXBOUNCE}",
            "%{SYSLOGBASE} %{POSTFIXQMGR}",
            "%{SYSLOGBASE} %{POSTFIXCLEANUP}"
        ]
        named_captures_only => true
    }
}

Since the patterns are quite complex, I’ve added them as a gist here. These are originally based on the rules from this gist, thankfully someone has already completed most of the hard work!

As any mail administrator / systems administrator knows, diagnosing issues with email is one of the least fun things to do. Especially in the case where there are dozens or even hundreds of mailservers with different configurations, this becomes a real nightmare. When we receive calls about mail not sending, instead of manually parsing logs (which could take 10 minutes or more) we can now run a search within 30 seconds. Further to this, we can create specific views to help diagnose patterns or anomalies so that we can see issues before they become a real problem. The power of nearly indexed data!

Custom Metrics

While PRTG gives us a lot of data which is easily graphed and alerted, we also collect a lot of custom data which we use for looking at trends and other custom alerting. Areas such as our clustered storage performance, SMTP stats and similar data we collect and process with Graphite. This gives us the ability to collect any time series data and easily pull it out for generating graphs or further analysis. The advantage we have is a “zero config” deployment of stat collection, based on our deployment systems and what we can collect off our main compute nodes. This is what drives our Central stats as well as our own internal monitoring.  

Here’s what our systems look like:

Collectd acts as our statistics collection using some of the built in plugins as well as custom scripts which we’ve written specific for our environment. We use RabbitMQ as a queuing system as it allows us to also divert stats to other systems if we want to store to another database or set custom alerts. The queuing system scales quite well and it’s certainly reliable, so having it as an intermediate system hasn’t degraded the reliability or performance.

Then, have Grafana (based on Kibana) which displays the information in a neat dashboard like manner. This is what we use for our internal monitoring and continually create different dashboards for various monitoring. For those who are used to just using Graphite dashboards, this is a far neater and easier system to configure. It’ll even import your existing dashboards from Graphite, so it’s well worth the few minutes to setup. 

Conclusion

This concludes our series on our NOC configuration, hopefully there’s been some useful information for those who are implementing similar systems. Like any monitoring system, nothing stays stagnant so we’ll be constantly updating the displays, logging more data and choosing how we display this data.

There are a myriad of tools out there which can greatly assist with the “big data” problem that exists with any system, no matter what the size. With the ELK stack, you get a comprehensive logging and search system. With Graphite / Grafana, you get a very powerful metrics collection and display system. By utilising both, you can very quickly gain a great insight into your entire platform. Especially if you combine this with an orchestration system such as SaltStack, you can automate the deployment of the data collection and further reduce the administration burden. This is all without having to fork out money for software like Splunk, which at the volumes we log can add significant costs.

If there’s anything you want more detail on or you think I’ve missed, please just add a comment below and I’ll do my best to provide you with the required info!