We had initially designed the spouse relationship as mutual. Mutual relations will be an early subject of discussion especially if humans are a part of your conceptual domain.However, defining the graph data model and ETL process for our data, which was first designed 25 years ago according to legislative regulations that dates back to 19th century and have been subject to frequent updates, had its own story.įirst let’s look at some of the challenges in data modeling. If you already have a small or well-designed SQL data, it is relatively easy to construct a property-graph model and make the initial data load. After realizing that great potential in graph databases, it was no longer possible for us to even consider a citizen database not making use of a graph database. Constructing domestic / international migration routes, investigating causes and discussing consequencesįor instance, it would require four join operations on tables with more than 100 million rows to reach a close relative such as a cousin of a person, which would require a lot more effort for the developer and consume a lot more system resources compared to Neo4j.Identifying old/helpless citizens without any relatives living nearby.Determining heirs and calculating heritage shares.Providing comprehensive and online real-time ancestral trees for citizens.Shortly after initial POC practices, it became quite clear that representing citizen data as a graph database was the perfect solution for the following scenarios among many others: The Benefits of Taking Citizen Data into a Graph After that, it did not take long to come upon the native graph solution: Neo4j. In fact we had spent some time on key-value and column-family databases till one day we realized that we were trying to construct a graph using other NoSQL databases. It was 2-3 years ago when we hit the graph database concept and Neo4j, while we were looking for possible solutions for some big data analysis requirements. We have been dealing with census data for more than 15 years. Although they are reluctant to alter their online transaction processing systems, they can’t resist the appeal created by the power of graph computing, especially due to the fact that Neo4j has reduced previously complex and scary graph science to such simple and smart solutions (yeah Cypher rocks!). To be able to discover and use those patterns and connections for business purposes, information technology veterans increasingly use graph databases. With such a vast amount of data, the patterns and connections in data become as important as the data itself. The digital universe is doubling in size every two years and expected to reach 44 zettabytes by 2020. Why Do We Need ETL and Sync from SQL to a Graph Challenges in data modeling, export and import.Benefits of taking citizen data into a graph.Why do we need ETL and sync from SQL to a graph.In this post I want to talk about the challenges we faced during our graph journey and how we overcame them.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |