psql enhancement idea

[Uwe C. Schroeder]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1


Hi everyone,

I'm just doing a load of database transfers using pg_dump. Because I changed
the schema of quite some tables I have to dump full column inserts.
The normal -D dump doesn't use a transaction, and I can see that this
certainly shouldn't be the default. Nonetheless I think it might be a nice
feature if there is a optional switch that creates a BEGIN .... COMMIT block
around the inserts of each table's data. Full column inserts are at least 10
times faster if used in a transaction. Ok I understand that the transaction
bites big chunks out of the ressources - which wouldn't matter if you do that
on a development machine with plenty of everything and the tables don't have
billions of rows.

Thought's anyone ?

UC

- --
Open Source Solutions 4U, LLC 2570 Fleetwood Drive
Phone: [phone] San Bruno, CA 94066
Cell: [phone] United States
Fax: [phone]
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.3 (GNU/Linux)

iD8DBQFBeAhljqGXBvRToM4RAhHBAJ9pnOxF2owze5k1+qBsjg2BB39BuwCePdpJ
clYgE7K5TD7HYhhHy3JudHQ=
=Xpsr
-----END PGP SIGNATURE-----


---------------------------(end of broadcast)---------------------------
TIP 3: if posting/reading through Usenet, please send an appropriate
subscribe-nomail command to ***@postgresql.org so that your
message can get through to the mailing list cleanly

[Tom Lane]

Post by Uwe C. Schroeder
I'm just doing a load of database transfers using pg_dump. Because I changed
the schema of quite some tables I have to dump full column inserts.

I don't think that conclusion follows from that premise. In recent
pg_dump versions (any that use a column list with COPY, which I think is
7.3 or later) there is no fundamental disadvantage to using COPY; it
should be semantically equivalent to INSERT-with-column-list commands.

The only reason I can see for using the INSERT option anymore is if you
think that some of the row insertions might fail, and you don't want
that to stop the load altogether. If that's what you want, then you'd
certainly not want to wrap the load in BEGIN/END, so there doesn't seem
to be much point in adding an option to do it.

regards, tom lane

---------------------------(end of broadcast)---------------------------
TIP 2: you can get off all lists at once with the unregister command
(send "unregister YourEmailAddressHere" to ***@postgresql.org)

[Uwe C. Schroeder]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Post by Tom Lane

Post by Uwe C. Schroeder
I'm just doing a load of database transfers using pg_dump. Because I
changed the schema of quite some tables I have to dump full column
inserts.

I don't think that conclusion follows from that premise. In recent
pg_dump versions (any that use a column list with COPY, which I think is
7.3 or later) there is no fundamental disadvantage to using COPY; it
should be semantically equivalent to INSERT-with-column-list commands.
The only reason I can see for using the INSERT option anymore is if you
think that some of the row insertions might fail, and you don't want
that to stop the load altogether. If that's what you want, then you'd
certainly not want to wrap the load in BEGIN/END, so there doesn't seem
to be much point in adding an option to do it.
regards, tom lane

The reason is that I made changes to the schema, i.e. changed the ordinal
position of columns and added some columns. Since the positioning of the
columns isn't the same anymore a copy will fail. Column Inserts will leave
the new columns untouched (set them to NULL or whatever the default) but can
handle the change in order of the columns (ok, the order thing is purely
cosmetic...).
I see your point that you don't want to fail the whole import when using full
inserts - which would happen in a transaction block. My point was basically
that I assume all inserts will succeed, however I can't use copy because of
the changes but doing it in a transaction will increase the insert speed by
at least 10 (chewing up memory certainly).

As a remark: normally I make pure data backups as copy and as full column
inserts. In case of a recovery I then can easily switch to single inserts if
the copy fails - eats a bit more diskspace but increases security (besides
the slony replication I'm running on top of that...)

Uwe

- --
Open Source Solutions 4U, LLC 2570 Fleetwood Drive
Phone: [phone] San Bruno, CA 94066
Cell: [phone] United States
Fax: [phone]
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.3 (GNU/Linux)

iD8DBQFBeDF0jqGXBvRToM4RAmrkAKCoRXnlmnRpp0DI5/cFeCyx97nm1QCfWJ/A
8zSHT4NXzG8LG7GFqpPGSbE=
=c99W
-----END PGP SIGNATURE-----


---------------------------(end of broadcast)---------------------------
TIP 4: Don't 'kill -9' the postmaster

[Tom Lane]

Post by Uwe C. Schroeder

Post by Tom Lane
I don't think that conclusion follows from that premise. In recent
pg_dump versions (any that use a column list with COPY, which I think is
7.3 or later) there is no fundamental disadvantage to using COPY; it
should be semantically equivalent to INSERT-with-column-list commands.

The reason is that I made changes to the schema, i.e. changed the ordinal
position of columns and added some columns. Since the positioning of the
columns isn't the same anymore a copy will fail.

No, it won't, not if you use a column list.

regards, tom lane

---------------------------(end of broadcast)---------------------------
TIP 7: don't forget to increase your free space map settings

[Uwe C. Schroeder]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Post by Tom Lane

Post by Uwe C. Schroeder

Post by Tom Lane
I don't think that conclusion follows from that premise. In recent
pg_dump versions (any that use a column list with COPY, which I think is
7.3 or later) there is no fundamental disadvantage to using COPY; it
should be semantically equivalent to INSERT-with-column-list commands.

The reason is that I made changes to the schema, i.e. changed the ordinal
position of columns and added some columns. Since the positioning of the
columns isn't the same anymore a copy will fail.

No, it won't, not if you use a column list.

Ahhh - I see what you mean, thanks for that hint.

A little remark then: the pg_dump manpage (7.4.3) states

--attribute-inserts
Dump data as INSERT commands with explicit column names (INSERT
INTO table (column, ...) VALUES ...). This will make restoration
very slow, but it is necessary if you desire to rearrange the
column ordering.

Maybe that should be fixed then. I was used to do the full insert statements
(ok, I've been using postgres long before it even had SQL) - maybe a old
habit then...

Uwe

- --
Open Source Solutions 4U, LLC 2570 Fleetwood Drive
Phone: [phone] San Bruno, CA 94066
Cell: [phone] United States
Fax: [phone]
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.3 (GNU/Linux)

iD8DBQFBeDW4jqGXBvRToM4RAu4+AJ9gchqGNrP/IUUYy8ySMWLyP6I0JgCdG1SH
h0RLcQppb2yLaTkrG/W5cvc=
=5tEF
-----END PGP SIGNATURE-----


---------------------------(end of broadcast)---------------------------
TIP 9: the planner will ignore your desire to choose an index scan if your
joining column's datatypes do not match

[Tom Lane]

Post by Uwe C. Schroeder
A little remark then: the pg_dump manpage (7.4.3) states
--attribute-inserts
Dump data as INSERT commands with explicit column names (INSERT
INTO table (column, ...) VALUES ...). This will make restoration
very slow, but it is necessary if you desire to rearrange the
column ordering.
Maybe that should be fixed then.

Mmm, you are right, that comment is a tad out of date. Patch committed
--- thanks for pointing it out!

regards, tom lane

---------------------------(end of broadcast)---------------------------
TIP 1: subscribe and unsubscribe commands go to ***@postgresql.org