I used this stored procedure to create a topology for all Census block groups in the contiguous US. It took almost 136 hours to run (on a fast, dedicated workstation - Ubuntu 23.04, PostgreSQL 16, 128 Gb RAM, 4Tb SSD - doing nothing else). Are there ways to speed up this kind of operation? I have a fairly well optimized postgresql conf file.
I didn't create a spatial index on the cbg.contig_block_groups_topo topo column and am now wondering if that would've helped. I noticed as more and more states were done, the slower it got. By the time I got to Wisconsin, for example, it took more than 10X California's time.
Also, would it help right after the COMMIT to run VACUUM ANALYZE on the table?
CREATE OR REPLACE PROCEDURE cbg.insert_bgs_by_state()
LANGUAGE 'plpgsql'
AS $BODY$
DECLARE
fips varchar;
BEGIN
FOR fips IN
SELECT unnest(array['01', '04', '05', '06', '08', '09', '10', '12', '13', '16', '17', '18', '19', '20',
'21', '22', '23', '24', '25', '26', '27', '28', '29', '30', '31', '32', '33', '34', '35', '36', '37',
'38', '39', '40', '41', '42', '44', '45', '46', '47', '48', '49', '50', '51', '53', '54', '55', '56'])
LOOP
RAISE NOTICE 'Processing state %', fips;
-- Attempt to insert data for the state
BEGIN
INSERT INTO cbg.contig_block_groups_topo (geoid, statefp, topo)
SELECT geoid, statefp, toTopoGeom(geom, 'contig_block_groups_topology', 1, 1)
FROM cbg.contig_block_groups
WHERE statefp = fips;
EXCEPTION
WHEN OTHERS THEN
RAISE NOTICE 'Error inserting data for state %: %', fips, SQLERRM;
CONTINUE; -- Continue to the next state
END;
-- Commit the transaction for each state
COMMIT;
END LOOP;
END
$BODY$;
group by statefp
instead of doing a loop ? Maybe there is a way to do that, and let postgres do the parallelization / optimization for you if it can...