The era of the cloud is providing us with new tools, which creates the potential for building new business models and disrupting the old ways of doing things. Which, depending on your viewpoint, is either a source of threat or opportunity. Possibly even both.
The database, as the name implies, will be the foundation of all these digital initiatives. This means database makers in the cloud era face a number of challenges, such as maintaining security and integrity. At the same time, changes bring new business opportunities. So, just what are these and how might Microsoft SQL Server 2016 be the best vehicle for meeting these new challenges?
Many companies are expanding into new, fast growing businesses in which the client base undergoes rapid expansion. The publishing industry exemplifies this trend, with new digital model publications expanding rapidly. For example, in the two years since its launch, the women’s lifestyle webzine Baby2Body managed to attract 470,000 subscribers in an already crowded market. The speed at which new systems grow cannot be predicted.
Corporate search is another growing technology that is becoming mission critical for enterprises because the intelligence within companies is not always readily accessible. With the workforces of global corporations becoming more mobile and the information they hold more disparate, there is a danger of highly paid executives duplicating their effort because people in, say, London are not aware that the expensive research they are about to commission has already been conducted by their counterparts in Boston.
Specialised search engines can scan tens of millions of servers (internally and externally) to find the intelligence needed by the workforce. This in turn creates a wealth of statistics, graphs and comparisons on various market trends. These search engines are often integrated with social media sites such as Facebook and LinkedIn too. Indeed, many search engines will power a number of other applications and be integrated with numerous dashboards.
In these cases companies need a database that will cater for all unexpected eventualities, whatever they may be. Consultants describe these qualities in terms of scalability, flexibility and reliability. As such, a database needs to be able to ramp up in size without losing its integrity, so that no matter how big and diverse and disparate the database may be, there is always only one version of the truth and a rapid response time. This means that as the volume of information expands, so can a proportionate amount of computing resources (such as processing power, memory, storage and networking) be apportioned to the system.
As corporate search saves so much money and time, it becomes a mission critical system and in turn the supporting databases undergo furious and unpredictable growth. As it becomes more popular, the number of users will soar too, something that could see the number of log-ins jumping from ten thousand to one million in a single day. As the popularity of a new system spreads, people rapidly
become reliant on it and feel powerless when the system goes down. In which case downtime is a disaster. So business continuity is a very important need.
In all the above cases the explosive growth of information that has to be held is an unpredictable event that must be catered for. According to IDC, 90% of the total data in the world today was created in the last three years. Meanwhile, hard disk drive storage has risen enormously. IBM’s first hard drive had a maximum capacity of 5MB, whereas a modern single hard disk drive can store 3TB. That represents a 62,914,000% increase. Under these circumstances, with data multiplying in volume and variety, the need for data science represents both a challenge and an opportunity.
Finance and TCO
Whether it’s a start up company or a new project within an enterprise, there is a much greater emphasis on keeping a limit on the Total Cost of Ownership (TCO). This makes SQL Server 2016 an important consideration as the database of choice, because users are able to customise and tailor solutions to their specific needs.
One of the important criteria for choosing SQL Server 2016 is the quality of its technical documentation. This is an important foundation that is often overlooked in the rush to build systems, but it’s a consideration whose importance is usually appreciated in the long term. Microsoft’s history of providing good technical support is something that many enterprises have to appreciate when projects evolve, as they invariably do, and complication sets in.
Key SQL Server 2016 strong points include its overall suitability for demanding web-based applications, its Geographic Information System (GIS) support and integration with programming language such as Perl.
Enterprises can create a search infrastructure that uses SQL 2016 to instantly normalise data as it rapidly feeds in from multiple diverse sources. One of the strengths of this system is its ability to turn a massive amount of unstructured data into structured data that can be stored, understood and searched by it users.
Another challenge is to find a system that can blend in the advantages of open source systems. It would be ideal to use, say, the Apache Solr open source search platform as front end, which would make use of its NoSQL features and transferring that data into SQL Server 2016. The necessary scalability and reliability is achieved by by using SQL 2016 in the cloud, working in partnership with a cloud services provider.
Companies can use Microsoft SQL Server 2016’s performance and built-in R Service to deliver advanced analytics over a hundred times faster than before. The secret sauce behind this is a parallelisation technique and a new improved system of memory management that makes for dramatically faster results on any platform.
One of the improvements created in the system is that analysts no longer have to move data out of the database in order to study it. This dramatically cuts the waiting time and brings more immediate insights. In a trial use of Microsoft Dynamics AX, a cloud-scale online ERP offering, a data scientist ran queries on a non-clustered columnstore index on transactional tables. The involvement of Microsoft SQL Server 2016 cut the aggregation times from hours to seconds.
Meanwhile, the hybridisation of enterprise computing continues apace. Some business applications – especially where data sovereignty, security and compliance are an issue – are best served by on premise systems. On the other hand, the flexibility that the cloud offers makes it easier for those systems where expectations and costs are variable and need a system that can accommodate instant changes, ramping up and down at will. The breadth of different computing platforms brings even more diversity to the modern IT landscape.
In order to cope with the technical diversity SQL Server 2016 has the tools to support cross-platform analytics. Polybase allows the user to run queries on external data in Hadoop or Azure blob storage. It can push computation to Hadoop where appropriate, so that your analytical application can join and integrate data from big data stores with its own data in its ‘relational store’. Microsoft R Services, which are integrated with SQL Server, can also run on multiple Hadoop distributions. Since this is integrated with Microsoft’s cloud service, Azure HDInsight, it gives the user the freedom of choice and the liberation that comes with a standard in developing analytics code. The system also allows users to develop analytical code using R Tools for Visual Studio.
The system’s immunity to disaster can be bolstered through a new feature in SQL Server 2106 that offers an Always-On availability group spanning on-premises and Microsoft Azure to provide high availability. It also provides disaster recovery for their data warehousing.
By allowing companies to offload queries to secondary replicas, the system can also achieve huge performance improvements. One early adopter claimed they enjoyed a ten-fold speed improvement on some of its extraction processes. “SQL Server 2016 in a hybrid-cloud environment with Azure really cuts through all the logistics of managing large data sets,” says Richard Sawicky, chief data officer at RealtyTrac.
Financial technology is another voracious user of databases as banking and financial services are becoming transformed by digital technology and many age-old processes are being re-invented.
Saxo Bank, for example, is an emerging name in global online trading. It says SQL Server 2016 will boost its performance by giving it much faster analysis, by changing the way that data is analysed. The system allows the bank to use an improved columnstore on a wider array of tables which allows for more fields to be searched at once. Using clustered columnstore indexes creates massive storage savings, cutting the amount of disk space needed for tables to a fifth of the original requirement. “We’ve seen queries run twice as fast with ten times the query performance after adopting columnstore,” says Francesco L’Erario, Saxo Bank’s data warehouse architect.
As the provision of services becomes more complicated, the underlying management needs to be fine tuned to be as simple as possible. This means integrating processes so there are as few moving parts as possible.
Human resources departments, for example, run ever more complicated searches in the quest for talent but they invariably need quick results. Real-time operational analytics in SQL Server 2016 will allow them to run analytical queries from the same data source. This makes the queries run ten
times faster with virtually no impact on the operational workload. The key to this is having an analytics database in the single data source, but which can by automatically updated by SQL Server.
Security As covered elsewhere in a companion piece, hacking and theft are a major concern. On this score Microsoft SQL Server already had a pretty respectable track record on security. The US National Institute of Standards and Technology (NIST), a government agency that monitors security vulnerabilities by technology, vendor and product, rated SQL Server as the database system with the least vulnerabilities of all major platforms. Now SQL Server 2016 has bolstered its security even further.
Always Encrypted helps protect data at rest and in memory, Transparent Data Encryption encrypts all user data now with low performance overhead, and Dynamic Data Masking (DDM) and Row Level Security (RLS) allow developers to build applications that restrict access and protect data.
Not only do companies need to keep their budget and capacity options open, the technical choices of the future cannot be easily predicted. This informs the design of databases like SQL Server 2016 where the designers have aimed to create a platform with options to meet most eventualities. When companies need to build their own supporting apps it caters for the most popular method for doing so with JSON support. Its Stretch Database cuts storage costs while keeping data available for querying in Azure. Also bundled in are data integration capacity with SQL Server Integration Services (SSIS), Data Quality Services (DQS) and Master Data Services (MDS). The package comes with data warehousing features closely integrated with analytics and machine learning through R. There’s also a Business Intelligence engine with Tabular and Multidimensional models.
Market analyst the Gartner Group identified SQL Server 2016 as “furthest in vision and ability to execute within the Leaders“ in its infamous Magic Quadrant analysis of the Operational Database Management System (DBMS) sector.
The way enterprises organise their computing resources – such as their processing engines, their memory, storage and networking – is being constantly changed by the evolution of the cloud industry. Meanwhile, the volumes, variety and velocity of changes in the data workload are creating massive challenges. It makes sense to position yourself to take full advantages of the opportunities this brings, if you use the best possible tools for the job. According to independent experts, the best platform for drilling down into these reserves of precious mines is SQL Server 2016.