How do you manage and store large volumes of geospatial data in your GIS system?

At the core geospatial data is just another column of data that contains location information. The rest of the data is the same as any other dataset. The only odd part is that some datasets we use happen to be images, or raster data. Since some data is derived from imagery of the earth, GIS tools have been built to work with the raster data rather than convert it to a columnar format. Vector data on the other hand is stored in a columnar format, the only difference is that it has a GEOMETRY column that contains points, lines (which are points connected by a line), and polygons (points that make up a shape). There are other types such as 3D data, LiDAR/point clouds, and more too.

i always manage my geospatial data in PostgreSQL + PostGIS, you can use them both database and map at the same location and easy to mange.

We can simplify geospatial data into two main types: vector and raster. Picture vector data as digital doodles on a map, points for spots, lines connecting them, and polygons creating shapes like city boundaries. On the other hand, raster data is like a grid of tiny pixels forming an image. Each pixel represents a small piece, making it ideal for things like satellite imagery or elevation maps. Whether drawing shapes or dealing with pixels, geospatial data is our language for understanding and exploring the world.

Quite general answer to a corresponding general question: Good practices of data normalisation implemented in a spatial database should be mitigating weight and slowness of large datasets. Depending on available budget, some tools and data formats are better performing than others (e.g.: I'm more comfortable with .gpkg than with .shp)

Geospatial data, encompassing information about Earth's locations and features, is categorized into vector and raster data types. Vector data, represented by points, lines, and polygons, depicts discrete entities like buildings or roads. Raster data, composed of pixels or grids, represents continuous surfaces like altitude or satellite imagery. Selecting the appropriate data type aligns with the application's specific requirements.

Public agencies are usually the most common source but more private organizations have been providing public versions of data such as Ookla, imagery providers like Maxar and Planet, Google public data, Microsoft building footprints, Uber, and others. These are generally made available on cloud storage for download or by the provider directly. There has been a boom in private providers creating high quality datasets for purchase in recent years.

There are vectors that can be in the form of Shapefile, KML, GeoJSON, GeoParquet etc. There are rasters that can be in the form of GeoTIFF, Cloud-Optimised GeoTIFF (COG). There are also nowadays different forms of Tilesets to server them more efficiently than before, such as MBtiles, PMTiles.

You can't just database up to a map. PERIOD. you have to ensure it's going to the correct coordinates. Any transfer of data from one GIS software to another requires detailed analysis of what was actually placed to ensure you're getting an appropriate analysis of WHERE those data points are. If you've loaded up you road data from a database and it's in the wrong projection,for instance, or coordinates not properly surveyed, the map will be inaccurate. Your road may be in the wrong county or township. And any data query will be off.

Depending on the needs, many municipalities are providing open data portals with varying degrees of data types (.shp, Geojson, API) and variety (boundaries, point locations, public facilities). Another method for some specific and local data is using either the Open Street Map web portal, or, the OSM Place Search plugin in Qgis. The data varies in quality and quantity and tends to be more robust for larger city areas. Also, ESRI's ArcOnline has some freely available data as well as using the "Add Data" finder/feature in ArcPro if using ESRI software. Finally, reaching out directly to specific municipal departments, university/college GIS departments or instructors and individuals via professional memberships such as URISA and Linkedin.

Geospatial data is readily available from a diverse range of sources, catering to various requirements in terms of type, scale, and quality. Public entities, such as national mapping agencies and census bureaus, often provide comprehensive datasets covering a wide range of topics and regions at minimal or no cost. Private companies and vendors, like Google and Esri, offer high-resolution, up-to-date information for specific domains or areas, but may charge for their services. Academic and research institutions focus on niche or specialized topics or regions, while user-generated or crowdsourced data from social media, mobile apps, and online platforms may provide geospatial data at no cost or in exchange for participation...

GIS does not mean you can replace "boots on the ground," forgive me, but DUH, stupid, verification of data by surveying is the only way to ensure any data is accurate. You will always need surveyors,geologists, and geographers,

Acquiring geospatial data involves various methods, each with its own advantages and considerations. Downloading from online repositories offers convenience and extensive data options, but requires reliable internet and storage. Importing from external devices provides flexibility and offline acquisition, but necessitates compatible hardware and software. Creating from existing data or scratch allows for customization and higher data quality, but demands time and expertise. The chosen method depends on project requirements and resource availability.

We can store, use and modify data seamlessly by ESRI Enterprise solutions. Whatever the size or wherever the data is stored, but we can access, use and modify data with scalable solutions using ESRI s infrastructure solutions especially in utility segment. Accessing from huge imagery now a days services from live satellite also available from Pixxel. This can be used for analysis and decision on day to day activities like weather prediction and making contingency plans. Can be also used in exploring minerals and other carbon hydro products.

For large volumes of vector data a geo-spatial database is the way forward. PostGIS is a good option with more functionality than MS SQL it is also free for none enterprise level. Currently favouring Databricks Mosaic or Apache Sedona. Both offer a lot of storage functionality but lack connectivity into mainstream GIS applications. Both can do GIS analysis at scale though so the only bit you lose is the visualisation of results. We use Mosaic a lot in our geo-pipelines, and it means our data can be stored and ready to be ingested into other products.

Are you asking a question or preaching to the converts? XML doesn't mean anything anymore? It used to be a wonderful way of embedding data in an image, raster or vector.

Managing and storing large volumes of geospatial data requires a structured approach. Organize data logically, document metadata and provenance, and choose appropriate formats and compression techniques. Utilize databases, such as relational or spatial, for efficient storage and querying, and consider servers for sharing data over the internet or a network.