3

I used this stored procedure to create a topology for all Census block groups in the contiguous US. It took almost 136 hours to run (on a fast, dedicated workstation - Ubuntu 23.04, PostgreSQL 16, 128 Gb RAM, 4Tb SSD - doing nothing else). Are there ways to speed up this kind of operation? I have a fairly well optimized postgresql conf file.

I didn't create a spatial index on the cbg.contig_block_groups_topo topo column and am now wondering if that would've helped. I noticed as more and more states were done, the slower it got. By the time I got to Wisconsin, for example, it took more than 10X California's time.

Also, would it help right after the COMMIT to run VACUUM ANALYZE on the table?

CREATE OR REPLACE PROCEDURE cbg.insert_bgs_by_state()
LANGUAGE 'plpgsql'
AS $BODY$
DECLARE
    fips varchar;
BEGIN
    FOR fips IN 
        SELECT unnest(array['01', '04', '05', '06', '08', '09', '10', '12', '13', '16', '17', '18', '19', '20', 
                            '21', '22', '23', '24', '25', '26', '27', '28', '29', '30', '31', '32', '33', '34', '35', '36', '37', 
                            '38', '39', '40', '41', '42', '44', '45', '46', '47', '48', '49', '50', '51', '53', '54', '55', '56'])
    LOOP
        RAISE NOTICE 'Processing state %', fips;
        
        -- Attempt to insert data for the state
        BEGIN
            INSERT INTO cbg.contig_block_groups_topo (geoid, statefp, topo)
            SELECT geoid, statefp, toTopoGeom(geom, 'contig_block_groups_topology', 1, 1)
            FROM cbg.contig_block_groups
            WHERE statefp = fips;
        EXCEPTION
            WHEN OTHERS THEN
                RAISE NOTICE 'Error inserting data for state %: %', fips, SQLERRM;
                CONTINUE;  -- Continue to the next state
        END;

        -- Commit the transaction for each state
        COMMIT;
    END LOOP;
END 
$BODY$;
3
  • 1
    you can try converting this script to python and using the multiprocessing library. you're also not using a spatial operation in a join or where clause so I'd assume a spatial index will not help you here
    – ziggy
    Commented Jan 3 at 17:49
  • How would mp help? I believe the topology between states is also desired? There are over 200k census block groups in total if I recall correctly. To me it makes sense that it takes longer the further in you are. I'm wondering if you could compute the topology individually for each state, then compute the topology for the neighbouring blocks at the borders of each state and afterwards fix the node/edge ids again... Commented Jan 3 at 20:56
  • Did you try to just do a group by statefp instead of doing a loop ? Maybe there is a way to do that, and let postgres do the parallelization / optimization for you if it can... Commented Jan 8 at 16:39

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.