postgresql crushed with XLogWrite error

[Tsirkin Evgeny]

Hello dear list!
I am using postgresql 7.3.3 .Last night the errors occured.
Since then I can't start the postgresql !We are recovering from the type
by now ,but what happened?Is that a known bug?What to do next time?
The last change I made was that:
syslog=1
log_min_error_statement
log_timestamp = true
stats_reset_on_server_start = false
I can't think why those changes could crush the postgres!

Here is the log:

#============================================================================================
2004-01-05 05:30:35 PANIC: XLogWrite: write request 15/EF40E000 is past
end of log 15/EF40E000
2004-01-05 05:30:35 LOG: statement: update tyuta_rashum set chugid=521
where studentid=3587261 and maslulsignid=8014022 and c
ourseid=67538;
2004-01-05 05:30:35 LOG: server process (pid 2527) was terminated by
signal 6
2004-01-05 05:30:35 LOG: terminating any other active server processes
2004-01-05 05:30:35 LOG: all server processes terminated;
reinitializing shared memory and semaphores
2004-01-05 05:30:35 LOG: database system was interrupted at 2004-01-05
03:42:11 IST
2004-01-05 05:30:35 LOG: checkpoint record is at 15/EF40DFC0
2004-01-05 05:30:35 LOG: redo record is at 15/EF40DFC0; undo record is
at 0/0; shutdown TRUE
2004-01-05 05:30:35 LOG: next transaction id: 88539506; next oid:
88761828
2004-01-05 05:30:35 LOG: database system was not properly shut down;
automatic recovery in progress
2004-01-05 05:30:35 FATAL: The database system is starting up
2004-01-05 05:30:35 FATAL: The database system is starting up
2004-01-05 05:30:35 FATAL: The database system is starting up
2004-01-05 05:30:35 LOG: ReadRecord: unexpected pageaddr 15/E640E000 in
log file 21, segment 239, offset 4251648
2004-01-05 05:30:35 LOG: redo is not required
2004-01-05 05:30:35 FATAL: The database system is starting up
2004-01-05 05:30:35 FATAL: The database system is starting up
2004-01-05 05:30:38 PANIC: XLogWrite: write request 15/EF40E000 is past
end of log 15/EF40E000
2004-01-05 05:30:38 LOG: startup process (pid 2529) was terminated by
signal 6
2004-01-05 05:30:38 LOG: aborting startup due to startup process failure
#============================================================================================

And since that, every time I am trying to start the db I get this:

#============================================================================================
2004-01-05 08:49:10 LOG: database system shutdown was interrupted at
2004-01-05 05:30:35 IST
2004-01-05 08:49:10 LOG: checkpoint record is at 15/EF40DFC0
2004-01-05 08:49:10 LOG: redo record is at 15/EF40DFC0; undo record is
at 0/0; shutdown TRUE
2004-01-05 08:49:10 LOG: next transaction id: 88539506; next oid:
88761828
2004-01-05 08:49:10 LOG: database system was not properly shut down;
automatic recovery in progress
2004-01-05 08:49:10 LOG: ReadRecord: unexpected pageaddr 15/E640E000 in
log file 21, segment 239, offset 4251648
2004-01-05 08:49:10 LOG: redo is not required
2004-01-05 08:49:12 PANIC: XLogWrite: write request 15/EF40E000 is past
end of log 15/EF40E000
2004-01-05 08:49:12 LOG: startup process (pid 2907) was terminated by
signal 6
2004-01-05 08:49:12 LOG: aborting startup due to startup process failure
#============================================================================================

Here is the ls -al from the pg_clog directory.
I can't see any problems in log file 21:

-rw------- 1 postgres postgres 262144 Jul 6 2003 0000
-rw------- 1 postgres postgres 262144 Jul 10 02:10 0001
-rw------- 1 postgres postgres 262144 Jul 16 00:24 0002
-rw------- 1 postgres postgres 262144 Jul 21 21:15 0003
-rw------- 1 postgres postgres 262144 Jul 27 11:53 0004
-rw------- 1 postgres postgres 262144 Jul 29 11:11 0005
-rw------- 1 postgres postgres 262144 Jul 30 18:41 0006
-rw------- 1 postgres postgres 262144 Aug 3 20:24 0007
-rw------- 1 postgres postgres 262144 Aug 5 02:07 0008
-rw------- 1 postgres postgres 262144 Aug 6 18:52 0009
-rw------- 1 postgres postgres 262144 Aug 11 17:51 000A
-rw------- 1 postgres postgres 262144 Aug 12 18:50 000B
-rw------- 1 postgres postgres 262144 Aug 14 02:34 000C
-rw------- 1 postgres postgres 262144 Aug 15 13:40 000D
-rw------- 1 postgres postgres 262144 Aug 17 18:04 000E
-rw------- 1 postgres postgres 262144 Aug 20 15:58 000F
-rw------- 1 postgres postgres 262144 Aug 24 10:09 0010
-rw------- 1 postgres postgres 262144 Aug 25 14:52 0011
-rw------- 1 postgres postgres 262144 Aug 27 15:15 0012
-rw------- 1 postgres postgres 262144 Aug 31 14:58 0013
-rw------- 1 postgres postgres 262144 Sep 8 10:05 0014
-rw------- 1 postgres postgres 262144 Sep 8 13:55 0015
-rw------- 1 postgres postgres 262144 Sep 8 23:46 0016
-rw------- 1 postgres postgres 262144 Sep 9 21:01 0017
-rw------- 1 postgres postgres 262144 Sep 10 02:00 0018
-rw------- 1 postgres postgres 262144 Sep 10 09:49 0019
-rw------- 1 postgres postgres 262144 Sep 10 12:16 001A
-rw------- 1 postgres postgres 262144 Sep 10 16:28 001B
-rw------- 1 postgres postgres 262144 Sep 11 04:51 001C
-rw------- 1 postgres postgres 262144 Sep 11 14:00 001D
-rw------- 1 postgres postgres 262144 Sep 14 13:57 001E
-rw------- 1 postgres postgres 262144 Sep 15 02:02 001F
-rw------- 1 postgres postgres 262144 Sep 15 10:41 0020
-rw------- 1 postgres postgres 262144 Sep 15 12:37 0021
-rw------- 1 postgres postgres 262144 Sep 15 15:19 0022
-rw------- 1 postgres postgres 262144 Sep 15 22:58 0023
-rw------- 1 postgres postgres 262144 Sep 16 13:26 0024
-rw------- 1 postgres postgres 262144 Sep 17 06:43 0025
-rw------- 1 postgres postgres 262144 Sep 17 15:27 0026
-rw------- 1 postgres postgres 262144 Sep 18 13:14 0027
-rw------- 1 postgres postgres 262144 Sep 21 13:37 0028
-rw------- 1 postgres postgres 262144 Sep 22 09:21 0029
-rw------- 1 postgres postgres 262144 Sep 22 15:44 002A
-rw------- 1 postgres postgres 262144 Sep 23 01:15 002B
-rw------- 1 postgres postgres 262144 Sep 23 10:49 002C
-rw------- 1 postgres postgres 262144 Sep 23 14:36 002D
-rw------- 1 postgres postgres 262144 Sep 24 00:32 002E
-rw------- 1 postgres postgres 262144 Sep 24 11:56 002F
-rw------- 1 postgres postgres 262144 Sep 24 22:18 0030
-rw------- 1 postgres postgres 262144 Sep 25 14:11 0031
-rw------- 1 postgres postgres 262144 Sep 29 12:43 0032
-rw------- 1 postgres postgres 262144 Sep 30 00:31 0033
-rw------- 1 postgres postgres 262144 Sep 30 18:49 0034
-rw------- 1 postgres postgres 262144 Oct 2 14:04 0035
-rw------- 1 postgres postgres 262144 Oct 19 08:42 0036
-rw------- 1 postgres postgres 262144 Oct 19 12:59 0037
-rw------- 1 postgres postgres 262144 Oct 19 16:11 0038
-rw------- 1 postgres postgres 262144 Oct 20 10:18 0039
-rw------- 1 postgres postgres 262144 Oct 20 14:40 003A
-rw------- 1 postgres postgres 262144 Oct 21 09:03 003B
-rw------- 1 postgres postgres 262144 Oct 21 15:06 003C
-rw------- 1 postgres postgres 262144 Oct 22 11:47 003D
-rw------- 1 postgres postgres 262144 Oct 22 23:09 003E
-rw------- 1 postgres postgres 262144 Oct 23 16:15 003F
-rw------- 1 postgres postgres 262144 Oct 26 11:09 0040
-rw------- 1 postgres postgres 262144 Oct 26 15:16 0041
-rw------- 1 postgres postgres 262144 Oct 27 11:00 0042
-rw------- 1 postgres postgres 262144 Oct 27 15:42 0043
-rw------- 1 postgres postgres 262144 Oct 28 10:45 0044
-rw------- 1 postgres postgres 262144 Oct 28 19:55 0045
-rw------- 1 postgres postgres 262144 Oct 29 13:31 0046
-rw------- 1 postgres postgres 262144 Oct 30 11:04 0047
-rw------- 1 postgres postgres 262144 Oct 31 15:34 0048
-rw------- 1 postgres postgres 262144 Nov 2 15:27 0049
-rw------- 1 postgres postgres 262144 Nov 3 16:16 004A
-rw------- 1 postgres postgres 262144 Nov 4 18:06 004B
-rw------- 1 postgres postgres 262144 Nov 6 08:59 004C
-rw------- 1 postgres postgres 262144 Nov 9 10:24 004D
-rw------- 1 postgres postgres 262144 Nov 10 12:55 004E
-rw------- 1 postgres postgres 262144 Nov 11 16:06 004F
-rw------- 1 postgres postgres 262144 Nov 12 19:03 0050
-rw------- 1 postgres postgres 262144 Nov 13 15:45 0051
-rw------- 1 postgres postgres 262144 Nov 20 12:14 0052
-rw------- 1 postgres postgres 262144 Dec 17 19:56 0053
-rw------- 1 postgres postgres 122880 Jan 5 00:08 0054





---------------------------(end of broadcast)---------------------------
TIP 3: if posting/reading through Usenet, please send an appropriate
subscribe-nomail command to ***@postgresql.org so that your
message can get through to the mailing list cleanly

[Andrew Sullivan]

Post by Tsirkin Evgeny
Hello dear list!
I am using postgresql 7.3.3 .Last night the errors occured.

^

Post by Tsirkin Evgeny
by now ,but what happened?Is that a known bug?What to do next time?

Yes, it's a known bug, and is the very reason 7.3.4 was released.
Upgrade.

A
--
----
Andrew Sullivan 204-4141 Yonge Street
Afilias Canada Toronto, Ontario Canada
<***@libertyrms.info> M2P 2A8
[phone] x110


---------------------------(end of broadcast)---------------------------
TIP 5: Have you checked our extensive FAQ?

http://www.postgresql.org/docs/faqs/FAQ.html

[Tsirkin Evgeny]

Thanks!
Now another two questions:
Is there any official bug report on this that i can show to
my boss?
Should i upgrade to 7.3.4 to fix that bug or it is better to get newly
version?
What i need is to restore my corrupted db from the tape ,upgrade postgresql
(say from rpm) and just start the server.I don't want to do all the dump -
restore
thing.

On Mon, 5 Jan 2004 08:49:13 -0500, Andrew Sullivan

Post by Andrew Sullivan

Post by Tsirkin Evgeny
Hello dear list!
I am using postgresql 7.3.3 .Last night the errors occured.

^

Post by Tsirkin Evgeny
by now ,but what happened?Is that a known bug?What to do next time?

Yes, it's a known bug, and is the very reason 7.3.4 was released.
Upgrade.
A

---------------------------(end of broadcast)---------------------------
TIP 8: explain analyze is your friend

[Andrew Sullivan]

Post by Tsirkin Evgeny
Thanks!
Is there any official bug report on this that i can show to
my boss?

Yes, it's in the release notes. IIRC, there were like 24 or 48 hours
between the releases because of this. Tom Lane occasionally beats
himself up about the bug on-list.

Post by Tsirkin Evgeny
Should i upgrade to 7.3.4 to fix that bug or it is better to get newly
version?

You can upgrade to the lastest 7.3.x release by just dropping the new
binary in place, IIRC -- read the release notes. In any case, you'll
need to get some version of 7.3 running first.

A

----
Andrew Sullivan 204-4141 Yonge Street
Afilias Canada Toronto, Ontario Canada
<***@libertyrms.info> M2P 2A8
[phone] x110


---------------------------(end of broadcast)---------------------------
TIP 4: Don't 'kill -9' the postmaster

[Tsirkin Evgeny]

Probably a dumm question ,but it is better to ask then be sorry.
Nothing is changed in config files ,so i am extractiing
the binary from rpm ,copy it to the place and start the server,right?
And thanks again!
On Mon, 5 Jan 2004 09:09:55 -0500, Andrew Sullivan

Post by Andrew Sullivan

Post by Tsirkin Evgeny
Thanks!
Is there any official bug report on this that i can show to
my boss?

Yes, it's in the release notes. IIRC, there were like 24 or 48 hours
between the releases because of this. Tom Lane occasionally beats
himself up about the bug on-list.

Post by Tsirkin Evgeny
Should i upgrade to 7.3.4 to fix that bug or it is better to get newly
version?

You can upgrade to the lastest 7.3.x release by just dropping the new
binary in place, IIRC -- read the release notes. In any case, you'll
need to get some version of 7.3 running first.
A
----
Andrew Sullivan 204-4141 Yonge Street
Afilias Canada Toronto, Ontario Canada
[phone] x110
---------------------------(end of broadcast)---------------------------
TIP 4: Don't 'kill -9' the postmaster

---------------------------(end of broadcast)---------------------------
TIP 9: the planner will ignore your desire to choose an index scan if your
joining column's datatypes do not match

[Andrew Sullivan]

Post by Tsirkin Evgeny
Probably a dumm question ,but it is better to ask then be sorry.
Nothing is changed in config files ,so i am extractiing
the binary from rpm ,copy it to the place and start the server,right?
And thanks again!

If you're upgrading via RPM, why not just do a regular RPM
installation? Doesn't that work? (Extracting binaries from RPMs and
doing other such things sounds like the way to break something.)

A
--
----
Andrew Sullivan 204-4141 Yonge Street
Afilias Canada Toronto, Ontario Canada
<***@libertyrms.info> M2P 2A8
[phone] x110


---------------------------(end of broadcast)---------------------------
TIP 8: explain analyze is your friend

[Tsirkin Evgeny]

On Mon, 5 Jan 2004 11:23:08 -0500, Andrew Sullivan
<***@libertyrms.info> wrote:

I want to keep my configs.Of course i can save my postgresql.conf
(and other files that i changed and don't currently remember)
to another location and then put it back in place of what rpm will
put ,but as i said ,i don't actually remember what i have changed :(

Post by Andrew Sullivan
If you're upgrading via RPM, why not just do a regular RPM
installation? Doesn't that work? (Extracting binaries from RPMs and
doing other such things sounds like the way to break something.)
A

---------------------------(end of broadcast)---------------------------
TIP 3: if posting/reading through Usenet, please send an appropriate
subscribe-nomail command to ***@postgresql.org so that your
message can get through to the mailing list cleanly

[Andrew Sullivan]

Post by Tsirkin Evgeny
On Mon, 5 Jan 2004 11:23:08 -0500, Andrew Sullivan
I want to keep my configs.Of course i can save my postgresql.conf
(and other files that i changed and don't currently remember)
to another location and then put it back in place of what rpm will
put ,but as i said ,i don't actually remember what i have changed :(

The only things that should make a difference to you should be in the
/etc directory, assuming the RPM follows the filesystem standard (I
believe it does). My suggestion is to back everything up, then
replace the RPM in the standard way, and then compare the files one
at a time to see what's different. I'll bet the binaries and
libraries, of course, and things under /etc/[postgres?]/, and that's
it. Most likely, it'll be postgresql.conf and pg_hba.conf, and maybe
pg_ident.conf.

A
--
----
Andrew Sullivan 204-4141 Yonge Street
Afilias Canada Toronto, Ontario Canada
<***@libertyrms.info> M2P 2A8
[phone] x110


---------------------------(end of broadcast)---------------------------
TIP 9: the planner will ignore your desire to choose an index scan if your
joining column's datatypes do not match

[Christopher Browne]

Post by Andrew Sullivan

Post by Tsirkin Evgeny
On Mon, 5 Jan 2004 11:23:08 -0500, Andrew Sullivan
I want to keep my configs.Of course i can save my postgresql.conf
(and other files that i changed and don't currently remember)
to another location and then put it back in place of what rpm will
put ,but as i said ,i don't actually remember what i have changed :(

The only things that should make a difference to you should be in the
/etc directory, assuming the RPM follows the filesystem standard (I
believe it does). My suggestion is to back everything up, then
replace the RPM in the standard way, and then compare the files one
at a time to see what's different. I'll bet the binaries and
libraries, of course, and things under /etc/[postgres?]/, and that's
it. Most likely, it'll be postgresql.conf and pg_hba.conf, and maybe
pg_ident.conf.

It's easy enough to check what files the RPM intends to update...

bash-2.05a$ rpm -q -p -l netpbm-devel-9.24-3.i386.rpm
/usr/include/pam.h
/usr/include/pammap.h
/usr/include/pbm.h
/usr/include/pbmshhopt.h
/usr/include/pgm.h
/usr/include/pm.h
... other files omitted ...

That can diminish how much is in the "everything" that needs to be
backed up...

[Tsirkin Evgeny]

On Mon, 5 Jan 2004 12:39:20 -0500, Andrew Sullivan

Post by Andrew Sullivan
The only things that should make a difference to you should be in the
/etc directory, assuming the RPM follows the filesystem standard (I
believe it does).

It does not ,rpm puts the staff in /var/lib

Post by Andrew Sullivan
My suggestion is to back everything up, then
replace the RPM in the standard way, and then compare the files one
at a time to see what's different. I'll bet the binaries and
libraries, of course, and things under /etc/[postgres?]/, and that's
it. Most likely, it'll be postgresql.conf and pg_hba.conf, and maybe
pg_ident.conf.
A

Well, this is what i eventually did ,it was just more convinient to
wget the rpms ,type rpm -Uhv postgres*rpm and then replace 2 files.
(It took some work to recall what i have changed in the config
files,thought)
That was simpler then rpm2cpio etc... as i thought at the beginning.
The server works fine by now,hope this XLogWrite thing will not come
back .
Thanks for the help.
Evgeny




---------------------------(end of broadcast)---------------------------
TIP 6: Have you searched our list archives?

http://archives.postgresql.org