Archives
While any software development initiative has unique features, some situations recur so often that I feel like I should have a recording that I can play back the next time that same situation comes up. One of these is the “What,” “How,” and “When” of software development.
Projects get into trouble when it’s not clear who owns these critical decisions, and—perhaps more importantly—when the wrong person or function tries to own one or more of them. When the business people try to own the technical “how” of a project, you know you’re headed for trouble.
Similarly, when the technical people start designing end-user features (the “what”) without input from the users or the business, that often ends in disaster as well. And when either function tries to dictate “when” without regard to “what” or “how,” that spells trouble big-time.
Just the other day, I heard a business person say, “It’s obvious what they need to do—why can’t they just start coding?” Here the business person was saying, essentially, that the “what” is known (at least in their own mind), so the “how” should be obvious—meaning that engineering should just start doing it.
In such situations, unless the engineers are truly incompetent (rare), it’s very doubtful that the business person speaking actually understands either the “what” or the “how.” The engineers certainly do not, or they would indeed be coding.
Recommended reading: Software, the Last Handmade Thing
When a business person makes a statement like this, if he or she is in a position of sufficient power that the engineers do indeed “just start coding” even in the absence of clarity around the what or the how, the project rarely ends well. In particular, it rarely, if ever, delivers what the business person had in mind, when and how they wanted it.
And—you guessed it—it’s the engineers who generally get blamed for the failure, not the person who insisted they go ahead no matter what.
Projects work best when the business says “what,” the engineers say “how,” and the business and technical people negotiate jointly in good faith over “when.” Sometimes the “when” is fixed—for example, a trade show-driven launch date or an investor deadline. In that case, the business and technical people need to negotiate over the “what” and “how.”
Similarly, either the “how” or the “what” might be fixed—for example, because you are making modifications to an existing system and have limited technical options, or you have committed to deliver a certain feature. In this case, the “when” and the other of the three independent variables (either “what” or “how” respectively) need to be negotiable. Otherwise, a predictable failure—and/or development burnout—will occur.
Perhaps the most frequent issue is when a single person or function tries to own all three—the what, the how, and the when—telling engineering what they need to develop, how they are going to develop it, and when the project is to be delivered. Unless the person doing so is a universal genius—rare—this inevitably leads to problems.
I worked with Steve Jobs for four years at NeXT, and even he rarely tried to dictate all three. Two out of three he would try for—but rarely, if ever, all three (and then not for long). Steve would generally defer to engineering on the “how” and would often (though sometimes grudgingly) accommodate strong pushback on the “when.” While I’ve never worked with Elon Musk, I get the sense he also listens to a core team of engineers he trusts. Unless you consider yourself smarter than Steve Jobs and Mr. Musk, you should pause to reconsider your own actions when you try to dictate what, how, and when to your engineering team.
Another often-overlooked facet of this puzzle is the fact that all three activities require communication. Even if the “what” seems clear in your own mind, it still needs to be expressed in terms that the engineering team can understand. This process of ‘backlog elaboration’ nearly always reveals gaps in the clarity of the initial vision, even if it might have seemed ‘obvious’ to you. Similarly, the ‘how’ may be clear to your technical leads, but it still needs to be expressed in architecture diagrams, sequence diagrams, API specs, and other artifacts that communicate the technical vision to the engineering team.
Only when the “what” and “how” are expressed in sufficient detail can a reliable “when” be produced. The fact that the “what” is clear in our business person’s mind, or the “how” is clear in the mind of the architect, does not mean that the person’s vision could be successfully operationalized without further work. This is why “just start coding” reveals a real gap in understanding of how successful software projects are implemented.
All this can be really fast—even verbally and at the whiteboard in some cases. But in general, the more input and understanding you get from the people actually doing the work, the better your backlog and the more accurate your timeline will be.
A proper appreciation for the value of each ingredient (“what,” “how,” and “when”), combined with due respect for the roles of their proper owners, is the key recipe for successful software development.
More helpful resources:
Despite uncertainty around regulation, millions are already interacting inside the metaverse, a market Ernst & Young expects could contribute over $3 trillion to global GDP by 2031. With the metaverse poised to dramatically change how banks, insurance companies, and other financial institutions engage with customers, IT leaders are focused intently on the challenges and opportunities ahead.
Banking and financial services IT professionals gathered recently for an immersive, one-hour VR roundtable discussion in the metaverse, co-hosted by GlobalLogic and The CXO Institute. In The Future of Banking: Doing Business in the Metaverse, hosted by GlobalLogic CTO Steven Croke and facilitated by yours truly, participants took a deep dive into innovative next-generation banking and finance solutions on the horizon and questions on how banks will feed consumer needs for personalization, interaction, convenience, and security.
In this article, you’ll find the highlights from our session, including top questions surfacing in banking and finance organizations as each plans its metaverse roadmap – plus your personal invitation to join GlobalLogic’s Monthly Metaverse Meetups for banking leaders and innovators. Let’s begin by exploring the most pressing challenges and opportunities digital leaders face as metaverse and VR adoption gradually increase.
Why Metaverse Planning is on Banking & Financial Service Roadmaps
Changing consumer demographics and rapidly advancing VR technologies drive massive opportunities for forward-thinking brands, and banking is ripe for disruption. A recent GlobalLogic survey revealed that 90% of Gen Z are willing to turn to big tech and nonbanks for better and faster banking services, and most participants in that demographic had “no idea why” they’d go into a branch when most basic things can be done quicker and easier online.
The same survey found that 80% of Gen Z respondents felt there was insufficient advice available about banking and financial products and that they did not understand how things like mortgages were structured. Investing was a key theme across our research interviews, and most participants brought the idea up unprompted. Inflation, skyrocketing housing costs, and increasing volatility in the job market are weighing heavily on consumers’ minds, and freelancing in various forms is becoming more common.
For all their diverse needs across employment, banking, shopping, and entertainment, people are looking for more immersive, engaging, and personalized experiences. Increasingly, they’re finding those in the metaverse – particularly Gen Z and Millennials (spanning ages 14 to 40), around 40% of whom have already used VR technology in some way. According to Deloitte, close to 50% of this cohort say they spend more time interacting with others on social media than in real life. Further, Gartner predicts that by 2026, 25% of people will spend at least one hour a day in the metaverse for work, shopping, education, social, and/or entertainment.
This state of current affairs in which consumers are seeking out financial advice and services and increasingly doing so online ought to cause concern for banks, Croke shared with The Future of Banking participants. How will your business respond to an emerging group of consumers who do not feel they need banks or understand them properly and have little or no desire to enter a branch?
Metaverse Presents Opportunities for Education, Support & Customer Experience
The banking sector’s adoption of cryptocurrency and blockchain has increased significantly in recent years and will account for 4% and 4.5% of metaverse revenue in 2025, respectively. But beyond these earliest and best-known DeFi products, how will your bank build trust in the metaverse and make virtual interactions more compelling than the bricks-and-mortar equivalent?
Several banks are setting up lounges or virtual branches as an entry point to the metaverse and using the space to establish a presence and nurture customer relationships. Offering education, support, and advice on financial products in the metaverse can enable financial services brands to engage Gen Z even as VR banking matures.
HSBC, for example, purchased virtual real estate in The Sandbox to engage and connect with sports, e-sports, and gaming enthusiasts. Is this the right idea?
IT leaders attending The Future of Banking event had mixed feelings regarding virtual banking services. They expressed skepticism about the likelihood of adoption without a specific incarnation of virtual offerings that fires the customer’s imagination. Banks will need to give customers compelling reasons to go to the metaverse to complete actions they can already do with mobile banking applications or develop actions they cannot experience with mobile or web interfaces. The next biggest hurdle will be understanding what that will look like across the industry.
Transitioning to a VR Financial Services Mindset
For one institution, KB Kookmin Bank in South Korea, it meant creating a virtual branch where simple transactions, such as remittances, can be managed at a teller window.
“We’re already seeing several banks now setting up branches… they’re essentially providing lounges for users to go into those branches and try and make them, effectively, a place to get a conversation going with customers,” Croke shared. Roundtable participants were asked whether they see replicated real-world experiences as the model for transactions in the Metaverse.
One delegate, a CTO for a large insurance brand, said he felt that HSBC’s approach made more sense. “History is littered with examples of trying to replicate something in a new medium and it not working as well… doing something different in a different medium would probably be a more fruitful direction forward.”
Perhaps a hybrid approach would be easier than a metaverse-native experience? Banks may consider creating products that mimic something in the real world with a VR twin; for example, mortgage applicants could access and explore a digital twin of the property they’re considering.
IT leaders must also consider how metaverse-native experiences might be handled in the future. “If you buy a ticket to a concert in the Metaverse, why would you not purchase that with a payment product that is Metaverse native?” Croke said.
Exploring Options for Metaverse Finance & Banking Products
Even if product development is still far off on the long-term planning horizon, bank leaders should be thinking today about broadening their ideas of what financial products could look like within the metaverse context.
“We’re already seeing value items being created in the Metaverse,” Croke said. “We’re seeing collectibles being created. We’re seeing equities, we’re seeing art being created. How are these going to be financed? How can one purchase those products? And if you think about storage, where do we store that value?”
Delegates questioned whether we should expect to see a dual business model, with banks in the Metaverse handling cryptocurrencies transacted through the banks or Metaverse ATMs. We tend to think today that the metaverse will not be able to handle traditional banking products. However, as one delegate pointed out, we may see this change once banking finds a strong use case to drive initial adoption and create demand for more services. “I think it’s about starting with a very niche, single-use case that’s killer, and everybody wants to use it, but I think that’s yet to be found,” he said.
Visualizations offer an interesting way to explore the possibilities today. “If I want to have a 3D visualization of risk, rather than today’s 2D diagram, for example… in 3D, I can move stuff with my fingers and share that information with other traders,” one delegate shared. “I think that’d be very helpful. So, I visualize value addition. I think that’s a pivotal point where Metaverse can start adding value to existing processes.”
The Maturation and Growth of Decentralized Finance
The huge uptake in cryptocurrency and NFTs has led to a new virtual economy, even after the initial buzz died down. This is a borderless, secure, and fast environment in which DeFi enables financial transactions to be performed by entities directly using smart contracts without financial intermediaries.
Still, we’ve not yet reached a point of maturity where people feel comfortable undertaking a number of activities and transactions in the metaverse outside of Gen Z and gamers. We have this community of early adopters who are already quite demanding and discerning in their metaverse experiences alongside a far larger population still trying to wrap their minds around the possibility.
Whether adoption and user behavior drive regulation or increased regulation opens the door to greater adoption remains to be seen. The rise of central bank digital currencies and expressed desire from Singapore monetary (likely the furthest ahead at this point) raises many questions about DeFi and its impact on metaverse adoption and maturation. Even so, it is clear today that banks must prepare now to put their arms around this economy as the experiment and learn now so they can be best positioned to move fast and innovate as opportunities open up.
Final Thoughts & Continuing the Conversation
Internet users rely on multiple apps for authentication. But does the Metaverse require that we now own our digital identity? And if we move across multiple platforms, doesn’t a unique digital identity become a prerequisite?
How do we combat money laundering and fraud in a virtual environment where a criminal can open a crypto wallet, fund their wallet with cryptocurrency, and buy a parcel on a chosen metaverse platform? Once the parcel is bought, they can build a store to hold their NFTs and sell NFTs as a cover for illicit products in real life.
These are just a few of the metaverse questions and challenges facing IT leaders in banking worldwide right now.
GlobalLogic will continue the conversation in our monthly Metaverse Innovation Meetups beginning Friday, November 10, to be held in VR. Join us for ongoing discussions about:
- current trends in banking in the metaverse
- successes in the industry and lessons learned
- brainstorming prototypes that will help define the app that will ultimately successfully drive VR adoption
We’re growing a community of like-minded innovators and business leaders to talk through ideas and help move banking in the metaverse forward in real ways. Will you join us?
Click here to email me your request for an invitation to GlobalLogic’s Monthly Metaverse Meetup.
Digital product development can be a game-changer for organizations, in the ways it facilitates a seamless, software-driven user experience. It can provide insights on taking a user-centric approach to planning and developing digitally-driven solutions that delight users, create new lines of revenue, and scale with your growing business.
Consistently applying a data-driven approach to digital product development helps your organization uncover customer insights, identify market trends, and validate hypotheses that result in products that better meet customer needs and drive business growth. Moreover, continuously iterating based on real-time insights ensures the products you’ve invested in are sustainable and evolve with your customers’ needs.
In today’s world, organizations are accumulating and sitting on large volumes of data from an increasing number of systems and interfaces. However, this comes with its fair share of challenges, including (but not limited to) data quality and reliability, scalability and infrastructure, data privacy and security, and the growing talent and expertise gap. We’ll take a closer look at these key considerations and more, so you can achieve a more data-driven approach to digital product development.
1. Data Quality, Reliability & Governance
While the availability of vast amounts of data offers opportunities for valuable insights, it also introduces the risk of incomplete, inaccurate, or inconsistent data. Ensuring data quality and reliability is essential to leveraging the full potential of a data-driven approach.
Incomplete or missing data can result in incomplete or skewed insights, leading to flawed decision-making. Without reliable data, organizations risk basing their strategies on faulty assumptions or incomplete information.
Overcoming this challenge calls for robust data governance processes. This includes defining data standards, establishing data collection and storage protocols, and implementing quality checks. Data validation techniques, such as data profiling, outlier detection, and consistency checks, are crucial in identifying and rectifying data anomalies. Regular data audits and monitoring processes help maintain data integrity and reliability over time.
Additionally, organizations can employ automated data validation tools and techniques to streamline the process and ensure a higher level of data quality. These tools can flag data inconsistencies, identify missing values, and validate data against predefined rules or business requirements.
2. Scalability and Infrastructure
The ability to process and analyze large volumes of data is essential for effective digital product development. As organizations gather increasing amounts of data from diverse sources, scalability and infrastructure become critical factors in harnessing the full potential of this data.
Traditional systems and infrastructure may not be equipped to handle the velocity, variety, and volume of data that digital product development demands. Processing and analyzing massive datasets require powerful computing resources, storage capacity, and efficient data processing frameworks.
Investing in scalable infrastructure ensures organizations can handle ever-growing data volumes without compromising performance. Cloud-based solutions, such as scalable cloud computing platforms and storage services, offer the flexibility to scale resources up or down based on demand. This elasticity allows organizations to handle peak workloads during intense data processing and analysis periods while avoiding excessive costs during periods of lower activity.
Modern technologies like distributed computing frameworks, such as Apache Hadoop and Apache Spark, provide the ability to parallelize data processing across clusters of machines, improving processing speed and efficiency. These frameworks enable organizations to leverage distributed computing power to tackle complex data analytics tasks effectively.
Recommended reading: The Evolution of Data & Analytics Technologies
3. Data Privacy and Security
A strong focus on data privacy and security in digital product development helps organizations maintain compliance, protect sensitive data, and foster customer trust. This, in turn, allows for more effective data-driven decision-making and enables organizations to leverage the full potential of their data assets while mitigating the inherent risks.
It’s not a matter of if it will happen but when, as IBM reports that 83% of organizations will experience a data breach. Those using AI and automation had a 74-day shorter breach lifecycle and saved an average of USD 3 million more than those without.
Safeguarding customer information and maintaining trust is crucial in a data-driven approach. This data often includes sensitive and personal information about individuals, such as personally identifiable information (PII) or financial data. Protecting this data from unauthorized access, breaches, or misuse is of paramount importance.
Organizations must comply with data privacy regulations, such as the General Data Protection Regulation (GDPR) or the California Consumer Privacy Act (CCPA). These regulations outline guidelines and requirements for the collection, storage, processing, and sharing of personal data. Adhering to these regulations ensures that organizations handle customer data responsibly and legally.
Companies can implement encryption techniques to protect data at rest and in transit, access controls, and user authentication mechanisms. Conducting regular security audits and vulnerability assessments is also best practice. Supporting these initiatives requires a culture of data privacy and security awareness among employees. Training programs and clear communication channels can help employees understand their roles and responsibilities in protecting data and recognizing potential security risks.
4. Interpreting and Extracting Insights
Extracting meaningful insights from complex and diverse datasets is crucial for driving product innovation and success. However, this task can be challenging without the expertise of skilled data scientists and analysts to apply advanced analytical techniques and statistical models. These professionals possess the skills to navigate vast amounts of data, identify relevant patterns, and extract actionable insights that inform product development strategies.
Data scientists and analysts involved in digital product development must have a deep understanding of statistical analysis, data mining, machine learning, and visualization techniques. They should also possess domain-specific knowledge to contextualize the data and derive meaningful insights relevant to the product and its target audience.
These professionals leverage analytical tools and programming languages to manipulate and analyze data, such as Python, R, SQL, and data visualization tools like Tableau or Power BI. They employ exploratory data analysis techniques, statistical modeling, predictive analytics, and other advanced analytical methods to uncover patterns, correlations, and trends within the data.
They can identify user behavior patterns, preferences, and pain points, allowing organizations to make data-driven decisions about feature enhancements, user experience improvements, and product roadmaps. Collaboration between data scientists, analysts, and product development teams is crucial for the successful interpretation and application of data insights.
And, of course, this leads us to…
5. Talent and Expertise Gap
Successfully blending software engineering and data analytics expertise enables organizations to build data-driven products that offer exceptional user experiences. However, bridging the talent and expertise gap by finding skilled professionals with a strong understanding of both disciplines can be a significant challenge.
Software engineers possess the technical prowess to design and build robust and scalable applications, while data analytics professionals can extract meaningful insights from data and apply them to inform product development strategies. The intersection of these skill sets is relatively new, and the demand for professionals who can bridge the gap is high. This creates a talent shortage and a competitive job market for individuals with software engineering and data analytics expertise.
To address this challenge, organizations must invest in talent acquisition strategies that attract individuals with hybrid skill sets. They can collaborate with educational institutions to develop specialized programs that equip students with the necessary knowledge and skills in both domains. Providing internships, training programs, and mentorship opportunities can also help nurture talent and bridge the expertise gap.
Organizations can foster cross-functional collaboration to encourage knowledge sharing between software engineering and data analytics teams. This allows professionals from different disciplines to learn from each other and leverage their collective expertise to drive innovation in digital product development.
Additionally, promoting a culture of continuous learning and professional development is crucial. According to McKinsey, which takes regular pulse checks of product-development senior executives, 53% of decision-makers believe skill building is the most useful way to address capability gaps, ahead of hiring, talent redeployment, and contracting in skilled workers. Encouraging employees to enhance their skills through training programs, industry certifications, and participation in conferences and workshops helps keep them updated with the latest advancements in software engineering and data analytics.
Recommended reading: A Digital Product Engineering Guide for Businesses
6. Data Integration and Compatibility
Integrating and compatibility between disparate data sources and systems is a major challenge for organizations. Establishing seamless data integration pipelines and ensuring system compatibility is crucial for successful data-driven digital product development.
Organizations often have many data sources, including internal databases, third-party APIs, customer feedback platforms, social media platforms, and more. These sources can generate data in various formats, structures, and locations, making integrating and harmonizing the data effectively complex.
Legacy systems further compound the challenge. Older systems may have limited compatibility with modern data analytics tools and techniques. Extracting, transforming, and loading data from legacy systems for analysis can be cumbersome and time-consuming.
To address these challenges, organizations need to adopt a strategic approach to data integration, including:
- Data architecture and planning to develop a robust data architecture that outlines data flows, integration points, and data transformation processes. This architecture should account for different data sources, formats, and systems in the product development lifecycle.
- Data integration tools and technologies to simplify the integration of disparate data sources. These tools can help automate data extraction, transformation, and loading (ETL) processes, ensuring smooth data flow across systems.
- API and middleware integration, which can facilitate seamless integration between systems and data sources. APIs provide standardized interfaces for data exchange, allowing different systems to communicate and share data effectively.
- Data transformation and standardization. Data transformation techniques play a vital role in harmonizing data from different sources. Standardizing data formats, resolving inconsistencies, and ensuring data quality during the transformation process enables more accurate and reliable analysis.
- Modernization efforts to improve compatibility with data analytics tools and techniques. This digital transformation could involve system upgrades, adopting cloud-based solutions, or implementing data virtualization approaches.
7. Data Visualization and Communication
By using data visually to tell a story through charts, graphs, dashboards, and other interactive visual elements, organizations can distill complex information into intuitive and easy-to-digest formats. Data visualization is pivotal in effectively communicating complex data insights to non-technical stakeholders.
In its raw form, data can be overwhelming and difficult to comprehend for individuals without a technical background. Complex datasets, statistical analyses, and intricate patterns can easily get lost in rows of numbers or dense spreadsheets. This is where data visualization comes into play, allowing stakeholders to grasp the key insights and trends at a glance.
Effective data visualization relies on understanding the audience and tailoring the visual representations accordingly. Different stakeholders have varying levels of familiarity with data and different areas of interest. The visualizations should be designed to align with their needs, ensuring the right information is conveyed clearly and concisely.
There are several key principles to consider when designing data visualizations for effective communication, including simplifying complex data, a visual hierarchy that highlights important information, contextualization and relevant comparisons, interactivity, and compelling storytelling.
Recommended reading: 4 Best Practices to Guide IoT and Dashboarding Projects
8. Ethical Use of Data
The collection and analysis of vast amounts of data give rise to ethical considerations. As organizations harness the power of data to drive product development strategies, it is essential to uphold the highest standards of ethical conduct. This includes respecting user privacy, protecting sensitive information, and ensuring data usage complies with applicable laws and regulations.
Obtaining informed consent from users is essential. Organizations must be transparent about the data they collect, how it is used, and the measures in place to protect it.
Fairness is another crucial aspect of ethical data use, ensuring that the organization is using unbiased algorithms, models, and analytical techniques that do not discriminate against individuals or perpetuate societal biases. Proactively assess and mitigate potential biases in data collection, analysis, and decision-making processes to ensure fairness and equity.
Social responsibility is another guiding principle in data-driven product development. Advocate for the ethical use of data to address societal challenges, foster positive social impact, and avoid harm to individuals or communities. Consider the broader implications of data practices and determine how your organization can actively contribute to creating a responsible and inclusive digital ecosystem.
Implementing ethical data practices requires a comprehensive approach that includes clear policies, regular audits, and ongoing training for employees. It’s well worth getting right. Ethical data practices contribute to the long-term sustainability and reputation of organizations, while also aligning with broader societal expectations and regulatory requirements.
9. Cost and ROI
Implementing big data and analytics solutions in digital product development comes with significant upfront costs, including investments in infrastructure, tools, and talent acquisition. Organizations must carefully evaluate the return on investment (ROI) to ensure that the benefits derived from analytics initiatives outweigh the associated expenses.
While the costs of implementing big data and analytics solutions can be substantial, the potential benefits are equally significant. Leveraging data efficiently allows organizations to gain valuable insights, make informed decisions, and drive business growth. Research from The Business Application Research Center (BARC) shows that companies leveraging their data efficiently see an average increase in profitability of 8% and a 10% reduction in costs.
Begin by clearly defining the specific business objectives and key performance indicators (KPIs) your big data and analytics initiatives aim to address. This provides a basis for evaluating the impact and effectiveness of the investments made.
Conduct a thorough cost-benefit analysis to assess the potential returns and associated costs of implementing big data and analytics solutions. Consider both tangible and intangible benefits, such as improved decision-making, enhanced customer experience, and increased operational efficiency.
When investing in infrastructure, consider scalability to accommodate future growth and increasing data volumes. Cloud-based solutions offer the flexibility to scale resources based on demand, minimizing upfront infrastructure costs while providing the necessary capabilities to handle growing data requirements.
Establish mechanisms to measure and track the ROI of big data and analytics initiatives. You’ll need to regularly assess the impact on key business metrics, such as revenue growth, cost savings, customer satisfaction, and operational efficiency.
10. Continuous Learning and Adaptation
Staying current with the latest advancements, best practices, and industry trends is vital in digital product development, where technological advancements, new methodologies, and emerging opportunities drive constant evolution. To remain competitive and harness the full potential of data, thought leaders must foster a culture of continuous learning and adaptability within their organizations.
Encourage teams to pursue professional development opportunities. It’s important to allocate time and resources for training and learning activities and provide access to relevant educational resources to facilitate these programs. Give employees space and time to establish knowledge-sharing platforms and communities of practice to facilitate the exchange of ideas and encourage collaboration, as well.
Agile methodologies, such as Scrum or Kanban, are great for promoting iterative development and continuous improvement. Apply these methodologies to data analytics projects to enable teams to adapt quickly to changing requirements, incorporate feedback, and continuously learn from data insights and even failures.
Continuous learning should extend beyond the boundaries of data and analytics, as cross-disciplinary collaboration and combining data-driven insights with domain expertise can lead to more innovative approaches in digital product development. Developing data literacy across the organization is crucial, and empowers individuals to make informed decisions, contribute to data-driven discussions, and effectively communicate insights to drive organizational success. Advocate for understanding and interpreting data among all stakeholders, regardless of their roles or technical backgrounds.
Conclusion
Applying a big data and analytics lens to digital product development means taking a strategic, data-driven approach encompassing technical solutions, organizational cultural shifts, investment in talent and infrastructure, adherence to ethical principles, and a culture of continuous learning.
Yes, it’s a tall order. Working alongside an experienced digital engineering partner like GlobalLogic through ideation, design, development, testing, deployment, and ongoing maintenance can help. We help organizations unlock the true potential of their data and get to market faster with innovative, compliant digital products that drive business success.
Want to learn more? Contact the GlobalLogic team today and see what we can do for you.
In an era dominated by data-driven decision-making, valuable and actionable insights have never been more essential for business success. This need has led to the rise of data marketplaces as a revolutionary solution that connects data providers with data consumers. But what is a data marketplace, and how can you use them to your company’s advantage?
This blog dives into the essentials of Data Marketplaces – what they are, and the compelling reasons why integrating one might be a game-changer for your business.
What is a Data Marketplace?
A data marketplace is a platform that brings together data providers and data consumers, facilitating the buying and selling of data. It serves as a dynamic hub where data creators and consumers converge; where a variety of data products are listed and made available to potential buyers.
Data marketplaces typically host a diverse range of datasets from different sources and providers, spanning various domains and industries. They often offer features like search and filtering capabilities, allowing users to discover relevant datasets based on their specific needs. They may include additional functionalities such as rating and review systems, pricing models, and data preview options. They focus on creating a marketplace environment where users can explore and select datasets from multiple providers, promoting transparency, accessibility, and ease of data discovery.
Getting to Know the Data Marketplace Ecosystem
The data marketplace ecosystem typically consists of the following key stakeholders:
Data Providers
These are organizations or individuals who offer their data assets for consumption. For example, anonymized patient data exposed by healthcare organizations can be used by pharmaceutical companies and consumed by researchers for clinical trials and drug development. They can be data aggregators, data brokers, research institutions, or even individual users who possess valuable data.
Data Consumers
These are organizations or individuals who seek access to specific datasets for analysis, research, or business purposes. For example E-commerce platforms use this data to analyze user behavior and purchase history to offer personalized product recommendations, Pharmaceutical companies and researchers use patients data for their clinical trials and drug development
Platform Operators
These are the entities that develop, maintain, and operate the data marketplace platform. They provide the infrastructure, security measures, and services necessary for data providers and consumers to interact within the marketplace.
Data Governance Authorities
In most cases, data marketplaces may have data governance authorities or regulatory bodies that establish policies, standards, and compliance requirements for data exchange and usage. These entities ensure that data privacy, security, and legal considerations are upheld within the marketplace ecosystem. Data governance authorities help ensure that data within the marketplace is managed, protected, and used in a responsible and compliant manner.
Data Marketplace Benefits
The data marketplace facilitates the seamless exchange of data across various entities whether within an organization, across industries, or even beyond geographical boundaries. Think of it as an organized marketplace for data, where valuable insights and information are readily available. Here are some of its benefits:
Catalyst for AI and Analytics
Data fuels AI and analytics initiatives. A data marketplace provides a rich pool of data for training AI models and conducting advanced analytics.
Unlocking Data’s Potential
Your business generates a plethora of data – structured, unstructured, and everything in between. A data marketplace harnesses this potential by making data accessible to those who can derive value from it. It’s a catalyst for turning raw data into actionable insights.
Accelerating Innovation
In a data marketplace, different stakeholders can access diverse datasets. This fuels innovation as creative minds from various domains collaborate, leading to fresh perspectives and inventive solutions.
Efficient Resource Utilization
Rather than each department or team siloing their data, the data marketplace centralizes data resources. This streamlines data collection, avoids duplication, and optimizes storage costs.
Data Monetization
Data Marketplaces allow organizations to develop and execute a data monetization strategy. They can choose to sell raw data, derived insights, or data-driven services, depending on their business objectives. For example, Healthcare organizations can aggregate and anonymize patient data to sell to pharmaceutical companies and researchers for clinical trials and drug development. E-commerce platforms can analyze user behavior and purchase history to offer personalized product recommendations. They can also sell this data to third-party retailers or advertisers looking to target specific customer segments.
Agility in Decision-Making
Timely access to pertinent data fuels quick and well-informed decisions. Relevant data is just a few clicks away, eliminating bottlenecks caused by data retrieval.
Collaboration Beyond Boundaries
If your business operates on a global scale, a data marketplace bridges geographical gaps. Teams from different locations can effortlessly exchange data, fostering collaboration.
Enhanced Data Governance
A well-structured data marketplace enforces data governance policies. It ensures data quality, security, and compliance, thus maintaining integrity across the board.
Expectations and Use Cases
The rise of data marketplaces has created significant expectations and opportunities across various industries. Organizations can leverage data marketplaces to enhance their business intelligence capabilities, for example. They gain access to external datasets that complement their internal data, enabling them to generate comprehensive insights and make data-driven decisions.
Data marketplaces serve as valuable resources for researchers and developers, as well. They can access specialized datasets for scientific research, machine learning model training, and innovation, accelerating their projects and fostering collaboration.
This is why they have gained traction across various industries, empowering organizations to access and leverage valuable datasets for a wide range of applications. Industries such as finance, healthcare, marketing, and transportation can leverage data marketplaces to enhance their services. Users can access real-time data feeds, consumer behavior data, geospatial data, and other relevant datasets to improve customer experiences and drive innovation.
Here are a few practical examples of industries where data marketplaces are extensively used:
- Financial Services: Utilizing external data sources for risk assessment, fraud detection, and customer insights.
- Healthcare: Leveraging medical records, research data, and patient-generated data for personalized medicine and healthcare analytics.
- Retail and E-commerce: Using customer behavior data, market trends, and competitor insights for targeted marketing and business intelligence.
- Smart Cities: Integrating data from various sources to optimize city operations, traffic management, and resource allocation.
Data marketplaces are rapidly evolving and offer tremendous potential for organizations to tap into the power of external data assets. However, careful consideration of data quality, privacy, security, and compliance is essential to ensure the success and trustworthiness of these marketplaces.
Challenges and Security Considerations
While data marketplaces offer immense value, ensuring security and privacy is of paramount importance. These are among the challenges facing organizations:
Data Privacy: Data marketplaces must establish robust privacy measures to protect the sensitive information contained in the datasets. Compliance with data protection regulations, anonymization techniques, and secure data transmission protocols are critical to maintaining privacy.
Data Quality and Trust: Data marketplaces need to implement mechanisms to verify the quality and authenticity of datasets. This includes data validation processes, transparency in data provenance, and reputation systems that establish trust between data providers and consumers.
Secure Infrastructure: The marketplace platform itself must have robust security measures in place. This includes secure authentication and access controls, encryption of data at rest and in transit, regular security audits, and protection against cyber threats.
Examples of Data Marketplaces to Know
Data marketplaces have proven to be effective platforms for data exchange across diverse industries. Depending on the specific requirements and field of expertise, one can discover other platforms customized for your industry or use case. Here are some noteworthy real-world instances of successful data marketplaces:
AWS Data Exchange: Amazon Web Services (AWS) Data Exchange is a data marketplace that allows data providers to securely publish and monetize their data products. Data consumers can easily find, subscribe to, and use the data they need for various applications and analytics.
Microsoft Azure Marketplace: Microsoft Azure Marketplace offers a wide range of data products, including datasets, APIs, and machine learning models. It enables data consumers to discover and access data assets that can be integrated into their Azure-based applications and workflows.
Google Cloud Public Datasets: Google Cloud Public Datasets presents a dynamic data marketplace within the Google Cloud Platform, offering a diverse range of public datasets for analysis. Spanning various industries and disciplines, this platform empowers users to execute big data analytics and machine learning workloads without the complexities of data movement.
Snowflake Data Marketplace: The Snowflake Data Marketplace grants seamless access to live, ready-to-query datasets from various providers across multiple industries. This platform allows users to explore and utilize a diverse array of data without the need for data copying or movement, offering a convenient and efficient solution for data consumers.
Kaggle Datasets: Kaggle, a platform for data science and machine learning competitions, hosts a dataset repository where users can discover and download various datasets contributed by the community.
Quandl: Quandl is a data marketplace that offers a vast collection of financial, economic, and alternative datasets. It caters to financial professionals, data analysts, and researchers looking for historical and real-time data.
Experian Online Marketplace: Experian is a global information services company that offers a wide range of services including credit reporting, data analytics, and decision making solutions
Data.gov: Data.gov is a public data portal provided by the U.S. government, offering access to a wide range of open datasets from various federal agencies.
Datarade.ai: Data exchange marketplace and platform that connects data buyers with data providers. It serves as a marketplace for the exchange of various types of data, catering to businesses and organizations in need of data for analytics, research, and other purposes
The Future of Data Marketplaces
As the data-driven landscape continues to evolve, the future of data marketplaces holds immense potential to reshape industries, foster innovation, and democratize data access. Here is a glimpse into what lies ahead:
- AI-Driven Data Discovery
- Advanced AI algorithms will enable personalized data discovery, suggesting datasets based on user preferences and context.
- Smart search engines will enhance data accessibility, making it easier for users to find relevant information.
- Blockchain based Data Marketplace
- Blockchain technology can enhance data trust, transparency, and data lineage in Data Marketplaces.
- Edge Data Marketplace
- Data Marketplaces may extend to edge computing environments, offering data closer to where it’s generated.
- AI-Powered Data Monetization
- AI algorithms could assist in pricing and monetization strategies for data providers, optimizing revenue generation.
Looking ahead, the future of data marketplaces holds immense potential. Advanced AI algorithms will personalize data discovery, enhancing accessibility. Blockchain technology may enhance data trust and transparency. Data marketplaces may extend to edge computing environments, and AI-powered strategies could optimize data monetization.
These developments are poised to reshape industries, foster innovation, and democratize data access in the evolving data-driven landscape. Embracing the evolving landscape of data marketplaces is key to staying at the forefront of data-driven innovation and success.
Is a Data Marketplace Right for You?
Data marketplaces have emerged as transformative platforms that revolutionize the utilization of data assets, offering a powerful solution for businesses, researchers, and industries alike. These platforms enable the easy access, sharing, and monetization of diverse datasets, driving data-driven innovation, collaboration, and growth in the digital age.
They empower organizations to overcome challenges, unlock new insights, create revenue streams through data monetization, and foster a data-driven culture. Embracing the concept of a data marketplace allows organizations to position themselves for success in the data-driven era, leveraging data to drive growth, competitiveness, and strategic decision-making.
Are you considering incorporating a marketplace in your organization’s data strategy? GlobalLogic helps our client partners build end-to-end solutions that improve customer engagement, optimize operations, and bring innovative new products to market faster. Learn more about our Data & Analytics Services here.
More helpful resources:
- Accelerating Data Platform Adoption & Frictionless Use Across the Enterprise
- The Evolution of Data & Analytics Technologies
- Improving Digital Transformation Outcomes with Hyperautomation
One of the most exciting developments in healthcare is the emergence of Software as a Medical Device (SaMD) as a more convenient and cost-effective means to deliver superior care to the tens of millions of people worldwide who suffer from various health conditions.
SaMD is defined by International Medical Device Regulators Forum (IMDRF) as “software intended to be used for one or more medical purposes that is capable of running on general purpose (non-medical) computing platforms.” In layman’s terms, SaMD is regulated software — installed and operated on “off-the-shelf (OTS)” computing platforms like mobile phones, tablets, laptops, desktops, servers and/or the cloud — that aids in diagnosis, screening, monitoring, or treating physiological conditions. These SaMD applications cover a wide spectrum of clinical patient conditions, from diabetes management solutions to cloud applications that analyze and generate patient-related insights viewed via a clinician’s portal.
Over the past decade, there has been a major evolution in medical devices. Previously, the vast majority of the medical device feature set was resident in the device itself. However, this landscape has experienced a paradigm shift. With advancements in software engineering, these features, and functionalities in SaMD are re-partitioned taking advantage of software and hardware components readily available in the market. Integrating third-party OTS hardware, software and/or libraries and services within SaMD applications has created additional clinical value by optimizing patient care in a more efficient and cost-effective manner. Medical device manufacturers using OTS hardware can take advantage of commercial operating systems, third-party software & services, and the hardware advances in memory, computing power, connectivity, communications and screen technology.
SaMD applications are regulated in nature, and are required to follow the same set of standards that govern medical device software, including various ISO/IEC standards that have been embraced by global regulators, such as ISO 13485, ISO 14971, IEC 62304, and IEC 62366. Depending on the product, other standards may also apply.
SaMD applications often take the form of patient companion apps, so-called because they often serve as a patient’s primary link and interaction point with the medical device system. For example, a chronic pain sufferer can utilize a SaMD patient companion app to adjust the energy levels associated with an implantable neurological stimulator device based on the guard bands setup by their clinician.
For a person living with diabetes, the patient companion application is usually part of a distributed diabetes management system that integrates with and presents information from a wearable insulin pump, a CGM (Continuous Glucose Monitor) and Cloud Analytics, helping the patient better manage their glucose levels.
Traditional insulin pumps contained all of their therapy and diagnostic functionality on the pump. While they were clinically effective, it was difficult to integrate data from third-party CGM sensors, and was hard to connect and upload data to applications used by clinicians to support remote patients. Leveraging a patient companion app on a mobile phone enables this functionality to be integrated and delivered to the patient’s mobile phone, and thus takes advantage of riding on top of a commercial platform utilized by millions of people.
Another value proposition associated with the use of SaMD applications on OTS hardware is the cost advantage of re-partitioning functionality that historically was on proprietary hardware. This re-design and re-partitioning can minimize the size of the medical device and reduce the CoGS (cost of goods sold) by reducing the number of physical components. SaMD applications on OTS hardware can also improve system usability and garner greater acceptance by patients, clinicians and payors.
Many medical device manufacturers who have developed SaMD apps have gained market share and top-level revenue, and some are realizing even greater gains by offering specialized SaMD applications or subscription services. This direct revenue is projected to grow from $4.4 billion in 2021 to $8.2 billion in 2027.1
This all seems very positive, but how are companies managing the transition? Since SaMD patient companion apps are usually part of a distributed system, focusing on system engineering, system risks and design fundamentals are key to partitioning the functionality across multiple components. Another key is embracing a full lifecycle support plan that tracks hardware, software and services changes/updates, and re-releases applications as required. Finally, UIX design is critical because users expect the same level of design and implementation they experience with their everyday phone apps.
Now let’s consider how SaMD applications impact clinicians. With the incredible growth in the use of implantable and wearable medical devices over the past decade, and an aging population, clinicians and healthcare organizations are challenged with the increased volume of patients requiring periodic follow-ups. With the use of ubiquitous high-bandwidth connectivity provided through mobile phones, and a SaMD application providing data uploads to a central server or a cloud, patient data can be automatically analyzed against pre-defined guard bands or limits. If limits are exceeded, clinicians can be alerted to take the appropriate action.
For device manufacturers, the time is now to embrace patient companion applications
As mentioned, patient companion apps are projected to grow at a robust rate for the next several years — which means that device makers who have yet to embrace companion apps risk competitive displacement if they don’t course correct.
The good news? Companies just now embarking on patient companion app strategies can apply lessons learned from those who have gone before:
- Focus on utilizing the appropriate and defined ISO/IEC standards for all aspects of the product development of these applications
- Focus on applications that deliver tangible, verifiable clinical capability based upon science, and/or operational value (not just buzz)
- Leverage the UIX design adopted by iOS and/or Android as appropriate while adhering to the applicable standards
- Partner with design, engineering and technical experts who have significant experience in developing SaMD applications, taking advantage of the learnings generated by developing and supporting numerous SaMD apps
GlobalLogic, a Hitachi Group Company, is a leader in digital product engineering that helps clients design and build innovative products, platforms, and digital experiences by integrating our strategic design, complex engineering, and vertical industry expertise with Hitachi’s Operating Technology and Information Technology capabilities. We bring extensive digital engineering experience to help companies develop companion apps, with hundreds of successful projects brought to market over the last half decade.
The time is now to consider how leveraging SaMD patient companion app can be utilized by your company to help achieve your companies clinical and operational goals.
For more information on how GlobalLogic, a Hitachi Group Company, can help you better engage your customers, innovate within predictable budgets, and bring the next generation of companion apps to market in the shortest possible time, visit https://www.globallogic.com/services/industries/healthcare-life-sciences/
Editor’s note: This article was originally published in CXO Today on July 26, 2023.
CXOToday engaged in an exclusive interview with Avnish Singh, SVP – Head of Content Engineering, GlobalLogic.
Please elaborate on how Content Engineering is revolutionizing the corporate and major enterprise (CME) industry through enhanced collaboration and knowledge management.
Response: Major Enterprises have large amounts of data in silos that get created due to geography, business function, scale, and other reasons. Over the last several years, these enterprises have taken conscious and elaborate steps to make this data available to everyone across the organization. Content Engineering practice plays a pivotal role in bringing in technologies and data experts who understand the data, consolidating it onto a common platform, and enabling enhanced collaboration by making it more searchable and accessible.
An important aspect is to make this data easily searchable and bring in the ability for employees to access relevant information quickly. This can be achieved by applying high-quality data tagging and labeling techniques when setting up the common data platform. Improving the search and accessibility of the information across the organization enhances collaboration by ensuring that there is always a single source of truth, that has structured information on which the employees can collaborate.
The organized approach benefits large enterprises with dispersed systems as it helps in breaking the silos and drives better knowledge management, and faster decision-making. Furthermore, the advantages of well-organized data extend to market growth and customer service. Organizations with multiple product/service lines can provide a seamless experience to their customers through properly tagged and centrally accessed data.
This can help drive customer sentiment and hence retention. With the advent of generative AI, the use of content engineering teams becomes much more important. The data and domain experts will continue to enable organizations in creating their LLMs, to help power knowledge management and collaboration across the organization.
How does GlobalLogic distinguish itself from other companies in the Content Engineering sector, and what sets it apart in terms of innovative approaches and implementations?
Response: Our company DNA is product engineering, a capability that distinguishes us in our industry. This gives us a deep understanding of the complexities of the product lifecycle and its inherent dependencies on accurate and timely data. We recognize that such data is not merely incidental, but a crucial driver in shaping the customer’s experience and the organization’s evolution. The value it imparts is far-reaching, driving strategic decisions, refining product development, and propelling market positioning.
Our approach to content engineering is profoundly influenced by our understanding of data in the product lifecycle. We go beyond mere data management and strive to unleash its full potential in terms of usability, accessibility, and impact.
To us, data and content are not mere digits and letters but invaluable assets that can shape the trajectory of the organization and create rich, meaningful experiences for its customers. We ensure the integrity and validity of data at all stages of its lifecycle, from inception and collection to processing, storage, and deployment. Our stringent quality control measures guarantee the accuracy of data and the credibility of the content we present. By doing so, we ensure that our content is not just informative but also reliable, consistent, and geared toward delivering the intended impact.
Not only that, but our content engineering services also drive digital transformation for clients, covering concept to platform to insights. Data and content are vital in the product life cycle that helps in aligning their journey with product evolution, ensuring true engineering value. Our expertise ranges from content digitization to machine learning, enabling diverse digital platforms. Through partnerships, we’ve built a strong cross-functional lab, supporting design, development, and maintenance.
Additionally, we provide full lifecycle digital product development services to our customers covering requirement analysis, development, testing, and maintenance for completed customized solutions, deployment, and integration. These reflect across multiple aspects such as Talent Acquisition understanding, Operations & Process excellence, Competitive Pricing/Volume Discount/Innovation Fund, Content Localization and Multilingual capabilities, Data Security, and Adoption of Emerging technologies.
Could you share examples of notable projects or case studies where GlobalLogic’s expertise in Content Engineering has significantly enhanced customer experience and achieved tangible business outcomes?
Response: Some notable case studies that resulted in enhanced customer experience and tangible business outcomes for our customers:
Case Study 1 – Enhancement of Navigation Maps for a leading ride-sharing platform company
Challenges: Our client was using third-party commercial maps, which posed a few business challenges. The third-party maps were not designed and did not have all the features required for ride-hailing services. This led to a compromised experience for both drivers and customers due to routing and ETA issues. Additionally, maps service downtime directly impacted revenue, leading to skyrocketing costs as the business grew, adversely impacting the bottom line and margins.
Business Outcome: Due to these challenges, the customer engaged GlobalLogic to help create their maps. We quickly set up a core team that understood the unique requirements for map creation for the ride-hailing service. The team then delivered excellent quality maps for 7 countries, processing road geometry of 659,000KM (adding 217,000KM new roads) with an accuracy of 99.61% for road geometry and 99.70% for navigation.
This led to the enhanced customer experience and the customer experienced multiple benefits such as:
- Enhanced Customer and Driver Experience through the improvement of overall route planning, excellent accuracy of pick-up/drop-off locations, and reduced navigation errors.
- Increased business value for customers and drivers through enhanced routing efficiency through optimized routes, reduced travel time and costs.
- Expanded service coverage through the addition of new roads leading to access to new riders leading to business growth Elimination of 3rd party maps license and subscription costs leading to improved bottom line and margins Case Study.
2 – AI-driven remote detection of medical conditions for a leading healthcare provider
Challenges: The customer, a leading provider of nutrition and therapeutic health products, launched a dermatology product for remote assessment of various skin-related diseases. But given the remote nature, the diagnosis of the diseases was not very effective. Further, the doctor’s and patient’s session time was much longer as the diagnosis process was long.
To solve these challenges, the customer wanted to use AI to identify various skin ailments. However, they did not have the required training dataset for this purpose. They tried to use their teams, but the process was taking very long. This is when they engaged GlobalLogic to help train their AI model with an appropriate machine learning training dataset.
Business Outcome: We deployed a team of experts that included AI Content Engineering experts and Doctors with MD Dermatologist expertise. This team developed two machine learning training datasets. The first dataset was worked by AI content engineers who annotated the thousands of images provided by the customer to label (image quality, body part, skin type/tone Fitzpatrick scale, lesion detection) the ROI (region of interest). The team of doctors then did the ROI evaluation on this first labeled dataset to identify the skin disorders. The customer then used these two datasets to train their AI model with very good accuracy, making their product a great success in the market.
This led to tremendous customer experience improvements for both the doctors and the patients as the time taken during the session was brought down by more than 50% in many cases and the AI-assisted identification of diseases led to much better accuracy of remote identification of the skin disorders.
How does GlobalLogic maintain the quality and precision of the structured data it delivers through Content Engineering? Are there specific processes or methodologies in place to ensure accuracy?
Response: GlobalLogic follows very stringent quality processes containing both manual and automated quality workflows. This is to ensure the quality and precision of the structured data. The quality workflow structures are customized based on the client’s requirements and expected deliverables. Our standard workflow includes:
Data Validation: We implement comprehensive validation rules to ensure that data entered into the system meets predefined criteria. This includes format checks, range checks, and consistency checks to identify and reject invalid or inconsistent data.
Data Cleansing: Once the data validation process is completed, we then clean and correct data to remove errors, duplicates, and inconsistencies. Furthermore, we also use automated tools, and scripts to identify and fix issues such as misspellings, incomplete records, or incorrect formatting.
Recommended reading: Continuous Testing: How to Measure & Improve Code Quality
Documentation and Metadata: We maintain comprehensive documentation and metadata about customer structured data. This includes recording the source, meaning, and context of each data element. Clear documentation helps prevent misinterpretation and ensures accurate usage of the data.
Regular Auditing: Periodic audits of customer data are conducted to identify and rectify any inconsistencies, inaccuracies, or missing values. This involves comparing data across different sources, verifying data against known benchmarks, or performing statistical analyses to identify outliers or anomalies.
Quality Assurance System: GlobalLogic has an in-house solution for Quality Assurance which is tailored as per the customer requirements. This system can be used with any type of process workflow.
Regular Data Backups: Regular data backups are performed to ensure that in case of any data loss or corruption, we can restore the data to its previous state. This minimizes the risk of losing valuable information and allows customers to maintain the integrity of their structured data.
Continuous Improvement: Our focus remains on continuous monitoring and improvement of customer data management processes. Feedback from users is collected to promptly address any data quality issues, and we regularly review and update customer data quality procedures to adapt to changing requirements and emerging best practices.
What are the primary technologies and tools utilized by GlobalLogic in its Content Engineering solutions, and how do they contribute to providing comprehensive support to customers?
Response: We leverage multiple in-house content engineering solutions and third-party solutions to deliver services to our customers. These are divided into the following categories:
Data Extraction and Web Scraping: We have built our Web Scraping tools using Python, BeautifulSoup, and Scrapy for extracting structured data from websites.
Extract, Transform, Load (ETL): Our Inhouse ETL solution provides features for extracting, transforming, and loading structured data from various sources into a target database or data warehouse.
Optical Character Recognition (OCR): Leveraging third-party OCR tools such as Tesseract and PDFMiner helps to extract structured data from scanned documents or images by recognizing and converting text into machine-readable formats. We also have our in-house tool named Dark Data Solution.
Data Quality and Precision: OpenRefine (formerly Google Refine), Google Sheets (With Apps Script), and a few other tools leveraged by us to help in cleaning and standardizing structured data. These tools automate tasks like removing duplicates, correcting formatting issues, and reconciling inconsistencies.
Labeling, Annotation & Classification: GlobalLogic has built its tool named LabelLogic that caters to all types of training data requirements, for next-generation ML models, through labeling, annotation & classification.
We leverage multiple accelerators, including Project Management Tool, Data Collection App, SLA Management Tool, and Auto Redaction of PI, while custom developing additional accelerators as needed. Our expertise in various tools and technologies like Python, Scrapy, Selenium, AWS, Google Cloud, Docker, Git, and more, further enhances our capabilities in delivering efficient solutions.
More helpful resources:
Enterprises envision a cutting-edge new system as their envisioned future state; when the outdated system has been phased out, the novel system takes over, and legacy data is managed while seamlessly integrating new data. In a successful digital transformation, this new system garners widespread approval from the extensive target audience, too.
Sounds great, right? Unfortunately, this isn’t always a smooth process, and there’s no guarantee of a successful outcome. According to McKinsey, a staggering 70% of digital transformations end in failure. This statistic paints a concerning picture, particularly when we consider that a significant portion of these failures can be attributed to unsuccessful migration endeavors.
It’s no wonder business leaders tend to get the “heebie jeebies” – a slang term meaning a state of nervous fear and anxiety – when it comes to migration. Often, migrations suffer from poor planning or exceed their allotted timeframes. In this article, we explore four different types of migration and share strategies to alleviate these apprehensions and combat the factors that can interfere with a migration’s success.
(Note: Within the context of this article, migration encompasses more than just data transfer; it encompasses a comprehensive system transition.)
Types of Migration
First, let’s explore four types of migration your organization might consider as part of its digital transformation.
Conventional Data Migration
Conventional data migration involves exporting data from source systems into flat files, followed by the creation of a program to read these files and subsequently load the data into the target system. It represents a more compartmentalized approach, suitable for scenarios where the disparity between the source and target data schema is minimal and the volume of data to be migrated remains relatively modest.
Here’s a real-life scenario in which an online pet pharmacy enterprise transitioned from an existing pharmacy vendor to a new one.
The groundwork was meticulously laid out for the new system, complete with a switch poised to be activated once the new system was infused with data. During the new pharmacy vendor’s launch, a migration task involving approximately 90,000 prescriptions from the old vendor’s database to the new one awaited. While not an overwhelming data load, it was substantial enough to warrant a deliberate decision. Consequently, the choice was made to employ the conventional data migration method.
The data was extracted from the previous vendor and handed over to our team. We meticulously refined the information, converting it into a format compatible with the new vendor’s system for seamless import. This comprehensive procedure was practiced and refined over the span of several months. The planning was executed with exact precision, carefully scheduling both full data feeds and incremental data updates. To ensure meticulous execution, we crafted a release checklist that enabled us to monitor and manage every step of the migration journey. Remarkably, the entire process unfolded seamlessly, maintaining uninterrupted service for the online pet pharmacy store’s end users.
Recommended reading: Easing the Journey from Monolith to Microservices Architecture
Custom Migration
In some cases, a migration process can become exceptionally intricate, demanding the establishment of a dedicated system solely for this purpose. This specialized software system, crafted specifically for the migration endeavor, follows its own distinct lifecycle and will eventually be retired once its mission is fulfilled.
Within the dynamic realm of the online travel industry, one of our clients is gearing up for a monumental migration undertaking. The intricacy of the issue at hand and the sheer volume of data involved necessitated the adoption of a highly customized service.
This bespoke solution was designed with a singular objective: to stage and subsequently transfer the data to the new system at the precise moment of user activation.
The sheer scale of this migration project is staggering, with the number of records to be migrated reaching the monumental figure of 250 million.
The existence of diverse source systems stands out as a key driver behind the adoption of this distinctive migration approach. This tailor-made service functions as a robust engine, adeptly assimilating data from various sources and meticulously readying it for integration into the staging database. Subsequently, the shift to the new system becomes a seamless transition, executed during runtime upon activation request. This precision-engineered and finely tuned custom solution sets the stage for the client’s journey toward a more enhanced operational landscape.
Data Migration Aided by Technology
Now, let’s envision taking the conventional data migration process and enhancing it with the power of modern automation through cutting-edge technology stacks. Picture the benefits of having tools seamlessly handle error handling, retries, deployments, and more. The prospect of achieving migration with such automated prowess might appear enticingly straightforward. However, there’s a twist. The success of this approach hinges on meticulous planning and agility, qualities that tools can aid in monitoring but ultimately require the deft touch of a skilled practitioner.
Several cloud services can assist in automating the various steps of migration. While I’m leaning toward an AWS PaaS-first approach here, it’s important to note that other leading cloud providers offer equivalent tools that are equally competitive.
The key components within such a migration system include:
- AWS Glue: AWS Glue serves as a serverless data integration service, simplifying the process of discovering, preparing, and amalgamating data.
- AWS S3: AWS Simple Storage Service proves invaluable for storing all ETL scripts and log storage.
- AWS Secret Manager: AWS Secret Manager ensures secure encryption and management of sensitive credentials, particularly database access.
- AWS CloudWatch: CloudWatch Events Rule plays a pivotal role in triggering scheduled ETL script execution, while CloudWatch Logs are instrumental in monitoring Glue logs.
- AWS DMS: AWS Database Migration Service (AWS DMS) emerges as a managed migration and replication service, enabling swift, secure, and low-downtime transfers of database and analytics workloads to AWS, with minimal data loss.
With the utilization of these services, let’s delve into how we can effectively execute the migration process:
This presents a straightforward workflow, leveraging AWS Glue, to facilitate data transfer from source to target systems. A crucial requirement for the successful execution of this workflow is establishing VPC peering between the two AWS accounts. It’s worth noting that there could be instances where client infrastructure constraints hinder such access. In such cases, it’s advisable to collaborate closely with the infrastructure team to navigate this challenge.
The process unfolds as follows: data undergoes transformation and finds its place within the stage database. Once the data is primed for activation, it is then seamlessly transferred to the target system through the utilization of AWS DMS.
While these tools undoubtedly streamline our development efforts, it’s essential to grasp how to harness their full potential. This aspect represents the simpler facet of the narrative; the true complexity arises when we engage in data validation post-migration.
On-Premises to Cloud Migration
This migration is the epitome of complexity – a quintessential enterprise scenario involving a shift from on-premise servers to cloud servers. The entire process is facilitated by a plethora of readily available solutions proffered by cloud vendors. A prime example is the AWS Migration Acceleration Program (MAP), an all-encompassing and battle-tested cloud migration initiative forged from our experience migrating myriad enterprise clientele to the cloud. MAP equips enterprises with cost-reduction tools, streamlined execution automation, and a turbocharged path to results.
Our collaboration extended to a leading authority in screening and compliance management solutions, embarking on a transformative journey. Among the ventures undertaken for this partner was the formidable Data Migration and 2-Way Sync project. The essence of this endeavor was to engineer a high-performance two-way synchronization strategy capable of supporting both the existing features of the On-Premises solution and those newly migrated to a novel, service-oriented framework on Azure. Furthermore, this solution was compelled to gracefully manage substantial volumes of binary content.
Take a look at the tech stack used for this migration:
Our solution comprised these integral components:
- ACL: A legacy component tasked with detecting alterations within the on-prem database and subsequently triggering events that are relayed to the cloud.
- Upstream Components: These cloud-based elements encompass a series of filtering, transforming, and persisting actions applied to changes. They are meticulously designed to anchor the modifications within the entity’s designated domain in the cloud. Moreover, these components generate replication events that can trigger responsive actions as required.
- Replication Components: Positioned in the cloud, these components specialize in receiving the replication events. They then proceed to either store the data or execute specific actions in response to the received events.
- MassTransit: In scenarios where cloud-induced changes necessitate synchronization back to the on-prem database, MassTransit steps in. This tool plays a pivotal role in reading all events generated in the cloud, forwarding them to downstream components, thus orchestrating the synchronization of changes.
Collectively, these components form a coherent framework that orchestrates the intricate dance of data synchronization between on-premises and cloud-based systems.
The achievement of two-way synchronization hinged on the utilization of key features within our product. These components included:
- Table-to-Table Data Synchronization: Our solution facilitated seamless data synchronization between on-premise and cloud databases, or vice versa. This process was orchestrated via an event-driven architecture, ensuring a fluid exchange of information.
- Change Capture Service for On-Prem Changes: In cases where alterations occurred on the on-premise side, a change capture service meticulously detected these changes and initiated corresponding events. These events were then synchronized to the designated home domain, simultaneously triggering notifications for other domains to synchronize their respective data, if deemed necessary.
- Cloud-Initiated Changes and Data Replication: Conversely, when changes manifested in the cloud, our solution orchestrated their transmission to the on-premise data replication service. This was achieved through a streamlined event-driven approach.
While much ground can be explored in the realm of on-premise to cloud migration, ongoing innovation, such as the integration of tools like CodeGPT, is consistently expanding the avenues for executing migrations. However, to stay focused on the core subject matter at hand, let’s get into the tips that can help alleviate the anxieties associated with these migration endeavors.
Tips for Migration Success
How can you ensure your next migration is successful? Don’t miss these crucial opportunities to simplify and combat the complexities of your migration.
1. Plan for Shorter and Early Test Cycles
Just as integrating and commencing testing early is pivotal in microservices architecture, kickstart the migration process early within the testing cycle. Incorporate numerous testing cycles to optimize the migration process. Our recommendation is to embark on five or more testing cycles. It’s of utmost importance that these cycles unfold in near-real-time production-like environments, replicating data closely resembling the production setting. Morphing tools can be employed to transplant sanitized production data into a staged environment.
Recommended reading: Continuous Testing – How to Measure & Improve Code Quality
2. Formulate a Comprehensive Validation Strategy
Leave no stone unturned when validating the migrated data. Thorough validation is essential to prevent financial losses or the risk of alienating customers due to a subpar post-migration experience. Here is an exemplary set of validation steps tailored for the post-migration scenario:
3. Initiate with Beta Users
Start the migration process by selecting a group of Alpha and Beta users who will serve as pilots for the migrated data. This preliminary phase aids in minimizing risks in the production systems. Handpick Alpha and Beta users carefully to ensure a smooth transition during live data migration. Alpha users constitute a smaller subset, perhaps around a hundred or so, while Beta users encompass a slightly larger group, potentially comprising a few thousand users. Eventually, the transition is made to a complete dataset of live users.
4. Anticipate Poison Pills
From the outset, plan for poison pills – records in Kafka that consistently fail upon consumption due to potential backward compatibility issues with evolved message schemas. Regularly checking for poison pills in production is a proactive measure to avert last-minute obstacles. Here’s a workflow that illustrates how to address poison pills:
5. Craft a Robust Rollback Strategy
Collaborate with clients to establish a comprehensive rollback strategy, ensuring that expectations are aligned. Conduct mock-run tests of the rollback strategy to preemptively address potential emergencies, as this could be the ultimate recourse to salvage the situation.
6. Seek Assistance When Available
If feasible, consider enlisting paid support to bolster your efforts. For instance, our client benefitted from licensed MongoDB support, utilizing allocated hours to enhance system performance and migration scripts. Such support often introduces a fresh perspective and intimate knowledge of potential challenges and solutions, making it invaluable during the migration process.
7. Incorporate Early Reviews
Be proactive in seeking reviews of the migration architecture from both clients and internal review boards. This diligence is vital to identify any potential roadblocks or discrepancies before they pose real-world challenges. By preemptively addressing issues raised during reviews, you can avoid last-minute complications, such as instances when a migration plan contradicts client policies, necessitating adjustments and improvements.
Conclusion
The vision of a seamless transition to a cutting-edge new system is an alluring prospect for enterprises, promising improved efficiency and enhanced capabilities. However, the journey from outdated systems to a technologically advanced future state is often fraught with challenges, and the alarming statistic that 70% of digital transformations end in failure, as highlighted by McKinsey, is a stark reminder of the complexities involved. Among the key contributors to these failures are unsuccessful migration endeavors, which underscore the critical importance of addressing migration apprehensions.
Indeed, the term “heebie jeebies” aptly encapsulates the anxiety that often accompanies migration processes. The anxiety can be attributed to a range of factors, including poor planning, exceeded timeframes, and unexpected roadblocks. Yet, as this article has explored, there are proven strategies to counter these challenges and achieve successful migrations. By embracing approaches such as shorter and early test cycles, comprehensive validation strategies, staged rollouts with Beta users, preparedness for potential obstacles like poison pills, and crafting effective rollback plans, enterprises can greatly mitigate the risks and uncertainties associated with migrations. Seeking expert assistance and incorporating early reviews also play crucial roles in ensuring a smooth migration journey.
The diverse types of migration covered in this article, from conventional data migration to custom solutions and on-premise to cloud transitions, demonstrate the range of scenarios and complexities that organizations may encounter. By diligently adhering to the strategies outlined here, enterprises can navigate the intricate dance of data synchronization and system transitions with confidence. As the digital landscape continues to evolve, embracing these best practices will not only help ease the “heebie jeebies” but also pave the way for successful digital transformations that empower organizations to thrive in the modern era.
Reach out to the GlobalLogic team for digital advisory and assessment services to help craft the right digital transformation strategy for your organization.
More helpful resources:
In planning a digital transformation, the CTO of an organization has many decisions to make to reach the final state. In order to achieve the overarching goal of sunsetting a legacy monolithic system, one such decision is whether to go with a brownfield or greenfield approach.
But that is the last stage, and we have a long road to travel to reach that place. There is a long lapse before the successful sunset, and it is perfectly acceptable to have the systems working midway. Typically, this means brownfield systems using much of the legacy monolith system, wrapped with a modern stack.
In this article, we’ll share guidance on easing the journey from monolith to a modern system, developing a North Star architecture, and how to adapt if needed at different stages of your digital transformation.
Getting Started: Evaluating Your Options
Once you’ve decided to modernize, there are different paths you can take: move to a completely new system, wrap/refactor your solution, or side-by-side.
Figure: Modernization Choices – GL POV
A lot of the thought leadership on this topic focuses on successfully transforming architecture and new systems. However, there is very little discussion about midway systems. If anything, people tend to talk about the older monolith/legacy systems and the amount of baggage they carry.
Recommended reading: Digital Transformation 101: Leveraging Technology to Drive Business Growth & Sustainability
But midways systems are complex and come with a lot of baggage. They can be labor-intensive and there are a lot of unknowns that can blow the budget. We’ve worked with many clients during this phase when their systems are midway and have learned important lessons to help make this a smoother process.
Many of these clients have started creating a greenfield system but modified the goal and decided to stay with the midway system for good. There are many reasons this may happen:
- External threats or change. The pandemic is a great example where organizations found themselves having to invest to keep up with an upsurge in orders during or post-pandemic.
- Lack of adoption. During testing or MVP rollouts, the client may struggle to gain acceptance and adoption from users.
- Cost factors. The budget runs out, and rather than having no system, clients stick to the half-prepared system, which still functions. This keeps the cash flowing, and the client decides to lose the battle to win the war at a later point in time.
We can indeed modernize the stack beneath before disrupting the actual user experience. This is when the entire monolithic system is not strangled. Seems like it is still alive while the engine underneath has been replaced. The harsh reality of such systems hits us hard – they are more complex than earlier and with more urgency to get to the other side.
The North Star Architecture
Moving towards North Star architecture is a Herculean effort and requires great perseverance. What’s more, a mid-way system needs more effort to maintain. So how do we move away from such a faux pas?
Here are the steps that have worked for us in the past. A visionary enterprise architect with a strong understanding of the old and new systems can help to come up with a blueprint to achieve this. Use these simple steps to chalk out your blueprint.
1. Continue with such mid-way systems with elan.
Since the mid-way systems are not going away too soon, try to make life more bearable with such systems. Invest in transition technology which will make mid-way systems simpler to operate and maintain.
We partnered with a global leader in veterinary practices software, products and professional services having $4B revenue, in creating a future state microservice based Global Prescription Management (GPM) platform for International expansion with current serving of over 10,000+ practices in North America & ~100K Global Supply Chain customers. This was a Greenfield approach to solve the current struggle of existing monolithic systems.
Recommended reading: Benefits of Total Experience (TX) Strategy in Modernizing Applications
The transition to the new system needs time and we cannot stop the business as usual. That is what funds the new system as well. So, we decided consciously to gear up for the new system with pipelines in place to migrate the data two ways. By “two ways,” we mean the data gets synced from new system to old and vice versa.
This essentially means that data is duplicated and maintained in sync for the sake of taking the step towards the transition to a new system. So, it looked something like this:
Figure: “Two-way” data sync
This pipeline will eventually sunset but it is worth the effort to keep the new system in sync with the old system and vice versa. The advantage we receive is a breather in our transition, when the enterprise looks at the more tasks in hand to move to a new system, adjusts the budget and timelines and overall makes life a bit easier because things are still running albeit with higher costs. The good news is the shop is still making money.
2. Build the orchestration layer.
The next step is to build an orchestration layer which will route the traffic from existing front end applications to the new backend systems. This layer will make sure users of the system will continue to have a seamless transition. In the background the old system has been replaced but users are still not impacted. Again this step can be executed by different deployment strategies like blue green deployment. And since the data is always synced in the background, you can make a switch back in case of any issues with the new system.
Figure: The Orchestrator Layer
While creating the set of APIs in the orchestration layer, architect it to be futuristic. The orchestration layer remains in the new system as well and is not a throw away code. For doing so, if there is a need for a wrapper layer or an adaptor for the legacy front end, let it be. This will be a throw away code but still will live its short life to glory. This is because it will keep the design for your orchestration clean and pristine.
3. Bite the silver bullet and migrate to a new system.
There is no easier way to do this than bite the silver bullet now. Create the new modern UI with your technology of choice and make the modern UI talk the same language as the orchestrator service. Phase out the old system in batches. Do an alpha test run, then select a cohort of beta users to migrate and finally the complete set of users can be migrated. This gives enough runway for the final launch.
Sometimes, migration to a new system requires more planning and effort than building greenfield applications. Such a big problem requires that we have to break it down into smaller problem statements and fix smaller problems one at a time. This may be as simple as on-premise to cloud migration and also may involve complex business rules to be applied for the transformation.
For instance, in our client’s case, we built a custom microservice whose sole responsibility was to ingest data from all disparate sources and place it in the final destination. This service had one time use but was easy to manipulate according to the business rules of transformation.
Figure: The final go live using custom Migration Engine
Conclusion
The monolith legacy system has lived its lives, and strangling the entire monolith is a humongous task. There are solutions to help ease the journey towards the North Star architecture. It is wise to invest in them early and use your budget to make lives simpler.
These mid-way system patterns are there to simplify your journey, and making use of them will help you avoid the faux pas of strangling the monolithic systems. GlobalLogic’s Digital Advisory & Assessment can help you in your journey from the monolith to a digitally transformed architecture to choose the right path, simplify steps along the way, and reach your end goal with a successful, sustainable solution to take your business forward.
More helpful resources: