source: trunk/src/sed/NEWS@ 3670

GNU sed NEWS                                    -*- outline -*-

* Noteworthy changes in release 4.9 (2022-11-06) [stable]

** Bug fixes

  'sed --follow-symlinks -i' no longer loops forever when its operand
  is a symbolic link cycle.
  [bug introduced in sed 4.2]

  a program with an execution line longer than 2GB can no longer trigger
  an out-of-bounds memory write.

  using the R command to read an input line of length longer than 2GB
  can no longer trigger an out-of-bounds memory read.

  In locales using UTF-8 encoding, the regular expression '.' no
  longer sometimes fails to match Unicode characters U+D400 through
  U+D7FF (some Hangul Syllables, and Hangul Jamo Extended-B) and
  Unicode characters U+108000 through U+10FFFF (half of Supplemental
  Private Use Area plane B).
  [bug introduced in sed 4.8]

  I/O errors involving temp files no longer confuse sed into using a
  FILE * pointer after fclosing it, which has undefined behavior in C.

** New Features

  The 'r' command now accepts address 0, allowing inserting a file before
  the first line.

** Changes in behavior

   Sed now prints the less-surprising variant in a corner case of
   POSIX-unspecified behavior.  Before, this would print "n".
   Now, it prints "X":

    printf n | sed 'sn\nnXn'; echo


* Noteworthy changes in release 4.8 (2020-01-14) [stable]

** Bug fixes

  "sed -i" now creates temporary files with correct umask (limited to u=rwx).
  Previously sed would incorrectly set umask on temporary files, resulting
  in problems under certain fuse-like file systems.
  [bug introduced in sed 4.2.1]

** Release

  distribute gzip-compressed tarballs once again

** Improvements

  a year's worth of gnulib development, including improved DFA performance


* Noteworthy changes in release 4.7 (2018-12-20) [stable]

** Bug fixes

  Some uses of \b in the C locale and with the DFA matcher would fail, e.g.,
  the following would mistakenly print "123-x" instead of "123":
    echo 123-x|LC_ALL=C sed 's/.\bx//'
  Using a multibyte locale or certain regexp constructs (some ranges,
  backreferences) would avoid the bug.  [bug introduced in sed 4.6]


* Noteworthy changes in release 4.6 (2018-12-19) [stable]

** Improvements

  sed now prints a clear error message when r/R/w/W (and s///w) commands
  are missing a filename. Previously, w/W commands would fail with confusing
  error message, while r/R would be a silent no-op.

  sed now uses fully-buffered output (instead of line-buffered) when
  writing to files. This should noticeably improve performance of "sed -i"
  and other write commands.
  Buffering can be disabled (as before) with "sed -u".

  sed in non-cygwin windows environments (e.g. mingw) now properly handles
  '\n' newlines in -b/--binary mode.

** Bug fixes

  sed no longer accesses invalid memory (heap overflow) when given invalid
  backreferences in 's' command [bug#32082, present at least since sed-4.0.6].

  sed no longer adds extraneous NUL when given s/$//n command.
  [related to bug#32271, present since sed-4.0.7]

  sed no longer accesses invalid memory (heap overflow) with s/$//n regexes.
  [bug#32271, present since sed-4.3].

** New Features

  New option, --debug: print the input sed script in canonical form
  and annotate program execution.


* Noteworthy changes in release 4.5 (2018-03-31) [stable]

** Bug fixes

  sed now fails when matching very long input lines (>2GB).
  Before, sed would silently ignore the regex without indicating an
  error. [Bug present at least since sed-3.02]

  sed no longer rejects comments and closing braces after y/// commands.
  [Bug existed at least since sed-3.02]

  sed -E --posix no longer ignores special meaning of '+','?','|' .
  [Bug introduced in the original implementation of --posix option in
  v4.1a-5-gba68fb4]

  sed -i now creates selinux context based on the context of the symlink
  instead of the symlink target. [Bug present since at least sed-4.2]
  sed -i --follow-symlinks remains unchanged.

  sed now treats the sequence '\x5c' (ASCII 92, backslash) as literal
  backslash character, not as an escape prefix character.
  [Bug present since sed-3.02.80]
  Old behavior:
     $ echo z | sed -E 's/(z)/\x5c1/' # identical to 's/(z)/\1/'
     z
  New behavior:
     $ echo z | sed -E 's/(z)/\x5c1/'
     \1


* Noteworthy changes in release 4.4 (2017-02-03) [stable]

** Bug fixes

  sed could segfault when invoked with specific combination of newlines
  in the input and regex pattern. [Bug introduced in sed-4.3]


* Noteworthy changes in release 4.3 (2016-12-30) [stable]

** Improvements

  sed's regular expression matching is now typically 10x faster

  sed now uses unlocked-io where available, resulting in faster I/O
  operations.

** Bug fixes

  sed no longer mishandles anchors ^/$ in multiline regex (s///mg)
  with -z option (NUL terminated lines). [Bug introduced in sed-4.2.2
  with the initial implementation of -z]

  sed no longer accepts a ":" command without a label; before, it would
  treat that as defining a label whose name is empty, and subsequent
  label-free "t" and "b" commands would use that label. Now, sed emits
  a diagnostic and fails for that invalid construct.

  sed no longer accesses uninitialized memory when processing certain
  invalid multibyte sequences. Demonstrate with this:
    echo a | LC_ALL=ja_JP.eucJP valgrind sed/sed 's/a/b\U\xb2c/'
  The error appears to have been introduced with the sed-4.0a release.

  The 'y' (transliterate) operator once again works with a NUL byte
  on the RHS.  E.g., sed 'y/b/\x00/' now works like tr b '\0'.  GNU sed
  has never before recognized \x00 in this context.  However, sed-3.02
  and prior did accept a literal NUL byte in the RHS, which was possible
  only when reading a script from a file.  For example, this:
    echo abc|sed -f <(printf 'y/b/\x00/\n')|cat -A
  is what stopped working. [bug introduced some time after sed-3.02 and
  prior to the first sed-4* test release]

  When the closed-above line number ranges of N editing commands
  overlap (N>1), sed would apply commands 2..N to the line just
  beyond the largest range endpoint.
  [bug introduced some time after sed-4.09 and prior to release in sed-4.1]
  Before, this command would mistakenly modify line 5:
    $ seq 6|sed '2,4d;2,3s/^/x/;3,4s/^/y/'
    1
    yx5
    6
  Now, it does not:
    $ seq 6|sed '2,4d;2,3s/^/x/;3,4s/^/y/'
    1
    5
    6

  An erroneous sed invocation like "echo > F; sed -i s//b/ F" no longer
  leaves behind a temporary file.  Before, that command would create a file
  alongside F with a name matching /^sed......$/ and fail to remove it.

  sed --follow-symlinks now works again for stdin.
  [bug introduced in sed-4.2.2]

  sed no longer elides invalid bytes in a substitution RHS.
  Now, sed copies such bytes into the output, just as Perl does.
  [bug introduced in sed-4.1 -- it was also present prior to 4.0.6]

  sed no longer prints extraneous character when a backslash follows \c.
  '\c\\'  generates control character ^\ (ASCII 0x1C).
  Other characters after the second backslash are rejected (e.g. '\c\d').
  [bug introduced in the sed-4.0.* releases]

  sed no longer mishandles incomplete multibyte sequences in s,y commands
  and valid multibyte SHIFT-JIS characters in character classes.
  Previously, the following commands would fail:
    LC_ALL=en_US.UTF-8 sed $'s/\316/X/'
    LC_ALL=ja_JP.shiftjis sed $'/[\203]/]/p'
  [bug introduced some time after sed-4.1.5 and before sed-4.2.1]

** Feature removal

  The "L" command (format a paragraph like the fmt(1) command would)
  has been listed in the documentation as a failed experiment for at
  least 10 years.  That command is now removed.

** Build-related

  "make dist" now builds .tar.xz files, rather than .tar.gz ones.
  xz is portable enough and in wide-enough use that distributing
  only .tar.xz files is enough.  It has been fine for coreutils, grep,
  diffutils and parted for a few years.


** New Features

  new --sandbox option rejects programs with r/w/e commands.


* Noteworthy changes in release 4.2.2 (2012-12-22) [stable]

* don't misbehave (truncate input) for lines of length 2^31 and longer

* fix endless loop on incomplete multibyte sequences

* -u also does unbuffered input, rather than unbuffered output only

* New command `F' to print current input file name

* sed -i, s///w, and the `w' and `W' commands also obey the --binary option
  (and create CR/LF-terminated files if the option is absent)

* --posix fails for scripts (or fragments as passed to the -e option) that
  end in a backslash, as they are not portable.

* New option -z (--null-data) to separate lines by ASCII NUL characters.

* \x26 (and similar escaped sequences) produces a literal & in the
  replacement argument of the s/// command, rather than including the
  matched text.

----------------------------------------------------------------------------
Sed 4.2.1

* fix parsing of s/[[[[[[[[[]//

* security contexts are preserved by -i too under SELinux

* temporary files for sed -i are not made group/world-readable until
  they are complete

----------------------------------------------------------------------------
Sed 4.2

* now released under GPLv3

* added a new extension `z` to clear pattern space even in the presence
of invalid multibyte sequences

* a preexisting GNU gettext installation is needed in order to compile
GNU sed with NLS support

* new option --follow-symlinks, available when editing a file in-place.
This option may not be available on some systems (in this case, the
option will *not* be a no-op; it will be completely unavailable).
In the future, the option may be added as a no-op on systems without
symbolic links at all, since in this case a no-op is effectively
indistinguishable from a correct implementation.

* hold-space is reset between different files in -i and -s modes.

* multibyte processing fixed

* the following GNU extensions are turned off by --posix: options [iImMsSxX]
in the `s' command, address kinds `FIRST~STEP' and `ADDR1,+N' and `ADDR1,~N',
line address 0, `e' or `z' commands, text between an `a' or `c' or `i'
command and the following backslash, arguments to the `l' command.
--posix disables all extensions to regular expressions.

* fixed bug in 'i\' giving a segmentation violation if given alone.

* much improved portability

* much faster in UTF-8 locales

* will correctly replace ACLs when using -i

* will now accept NUL bytes for `.'

----------------------------------------------------------------------------
Sed 4.1.5

* fix parsing of a negative character class not including a closed bracket,
  like [^]] or [^]a-z].

* fix parsing of [ inside an y command, like y/[/A/.

* output the result of commands a, r, R when a q command is found.

----------------------------------------------------------------------------
Sed 4.1.4

* \B correctly means "not on a word boundary" rather than "inside a word"

* bugfixes for platform without internationalization

* more thorough testing framework for tarballs (`make full-distcheck')

----------------------------------------------------------------------------
Sed 4.1.3

* regex addresses do not use leftmost-longest matching.  In other words,
  /.\+/ only looks for a single character, and does not try to find as
  many of them as possible like it used to do.

* added a note to BUGS and the manual about changed interpretation
  of `s|abc\|def||', and about localization issues.

* fixed --disable-nls build problems on Solaris.

* fixed `make check' in non-English locales.

* `make check' tests the regex library by default if the included regex
  is used (regex tests had to be enabled separately up to now).

----------------------------------------------------------------------------
Sed 4.1.2

* fix bug in 'y' command in multi-byte character sets

* fix severe bug in parsing of ranges with an embedded open bracket

* fix off-by-one error when printing a "bad command" error

----------------------------------------------------------------------------
Sed 4.1.1

* preserve permissions of in-place edited files

* yield an error when running -i on terminals or other non regular files

* do not interpret - as stdin when using in-place editing mode

* fix bug that prevented 's' command modifiers from working

----------------------------------------------------------------------------
Sed 4.1

* // matches the last regular expression even in POSIXLY_CORRECT mode.

* change the way we treat lines which are not terminated by a newline.
Such lines are printed without the terminating newline (as before)
but as soon as more text is sent to the same output stream, the
missing newline is printed, so that the two lines don't concatenate.
The behavior is now independent from POSIXLY_CORRECT because POSIX
actually has undefined behavior in this case, and the new implementation
arguably gives the ``least expected surprise''.  Thanks to Stepan
Kasal for the implementation.

* documentation improvements, with updated references to the POSIX.2
specification

* error messages on I/O errors are better, and -i does not leave temporary
files around (e.g. when running ``sed -i'' on a directory).

* escapes are accepted in the y command (for example: y/o/\n/ transforms
o's into newlines)

* -i option tries to set the owner and group to the same as the input file

* `L' command is deprecated and will be removed in sed 4.2.

* line number addresses are processed differently -- this is supposedly
conformant to POSIX and surely more idiot-proof.  Line number addresses
are not affected by jumping around them: they are activated and
deactivated exactly where the script says, while previously
    5,8b
    1,5d
would actually delete lines 1,2,3,4 and 9 (!).

* multibyte characters are taken in consideration to compute the
operands of s and y, provided you set LC_CTYPE correctly.  They are
also considered by \l, \L, \u, \U, \E.

* [\n] matches either backslash or 'n' when POSIXLY_CORRECT.

* new option --posix, disables all GNU extensions.  POSIXLY_CORRECT only
disables GNU extensions that violate the POSIX standard.

* options -h and -V are not supported anymore, use --help and --version.

* removed documentation for \s and \S which worked incorrectly

* restored correct behavior for \w and \W: match [[:alnum:]_] and
[^[:alnum:]_] (they used to match [[:alpha:]_] and [^[:alpha:]_]

* the special address 0 can only be used in 0,/RE/ or 0~STEP addresses;
other cases give an error (you are hindering portability for no reason
if specifying 0,N and you are giving a dead command if specifying 0
alone).

* when a \ is used to escape the character that would terminate an operand
of the s or y commands, the backslash is removed before the regex is
compiled.  This is left undefined by POSIX; this behavior makes `s+x\+++g'
remove occurrences of `x+', consistently with `s/x\///g'.  (However, if
you enjoy yourself trying `s*x\***g', sed will use the `x*' regex, and you
won't be able to pass down `x\*' while using * as the delimiter; ideas on
how to simplify the parser in this respect, and/or gain more coherent
semantics, are welcome).


----------------------------------------------------------------------------
Sed 4.0.9

* 0 address behaves correctly in single-file (-i and -s) mode.

* documentation improvements.

* tested with many hosts and compilers.

* updated regex matcher from upstream, with many bugfixes and speedups.

* the `N' command's feature that is detailed in the BUGS file was disabled
by the first change below in sed 4.0.8.  The behavior has now been
restored, and is only enabled if POSIXLY_CORRECT behavior is not
requested.

----------------------------------------------------------------------------
Sed 4.0.8

* fix `sed n' printing the last line twice.

* fix incorrect error message for invalid character classes.

* fix segmentation violation with repeated empty subexpressions.

* fix incorrect parsing of ^ after escaped (.

* more comprehensive test suite (and with many expected failures...)

----------------------------------------------------------------------------
Sed 4.0.7

* VPATH builds working on non-glibc machines

* fixed bug in s///Np: was printing even if less than N matches were
found.

* fixed infinite loop on s///N when LHS matched a null string and
there were not enough matches in pattern space

* behavior of s///N is consistent with s///g when the LHS can match
a null string (and the infinite loop did not happen :-)

* updated some translations

----------------------------------------------------------------------------
Sed 4.0.6

* added parameter to `v' for the version of sed that is expected.

* configure switch --without-included-regex to use the system regex matcher

* fix for -i option under Cygwin

----------------------------------------------------------------------------
Sed 4.0.5

* portability fixes

* improvements to some error messages (e.g. y/abc/defg/ incorrectly said
`excess characters after command' instead of `y arguments have different
lengths')

* `a', `i', `l', `L', `r' accept two addresses except in POSIXLY_CORRECT
mode.  Only `q' and `Q' do not accept two addresses in standard (GNU) mode.

----------------------------------------------------------------------------
Sed 4.0.4

* documentation fixes

* update regex matcher

----------------------------------------------------------------------------
Sed 4.0.3

* fix packaging problem (two missing translation catalogs)

----------------------------------------------------------------------------
Sed 4.0.2

* more translations

* fix build problems (vpath builds and bootstrap builds)

----------------------------------------------------------------------------
Sed 4.0.1

* Remove last vestiges of super-sed

* man page automatically built

* more translations provided

* portability improvements

----------------------------------------------------------------------------
Sed 4.0

* Update regex matcher

----------------------------------------------------------------------------
Sed 3.96

* `y' command supports multibyte character sets

* Update regex matcher

----------------------------------------------------------------------------
Sed 3.95

* `R' command reads a single line from a file.

* CR-LF pairs are always ignored under Windows, even if (under Cygwin)
a disk is mounted as binary.

* More attention to errors on stdout

* New `W' command to write first line of pattern space to a file

* Can customize line wrap width on single `l' commands

* `L' command formats and reflows paragraphs like `fmt' does.

* The test suite makefiles are better organized (this change is
transparent however).

* Compiles and bootstraps out-of-the-box under MinGW32 and Cygwin.

* Optimizes cases when pattern space is truncated at its start or at
its end by `D' or by a substitution command with an empty RHS.
For example scripts like this,

    seq 1 10000 | tr \\n \  | ./sed ':a; s/^[0-9][0-9]* //; ta'

whose behavior was quadratic with previous versions of sed, have
now linear behavior.

* New command `e' to pipe the output of a command into the output
of sed.

* New option `e' to pass the output of the `s' command through the
Bourne shell and get the result into pattern space.

* Switched to obstacks in the parser -- less memory-related bugs
(there were none AFAIK but you never know) and less memory usage.

* New option -i, to support in-place editing a la Perl.  Usually one
had to use ed or, for more complex tasks, resort to Perl; this is
not necessary anymore.

* Dumped buffering code.  The performance loss is 10%, but it caused
bugs in systems with CRLF termination.  The current solution is
not definitive, though.

* Bug fix: Made the behavior of s/A*/x/g (i.e. `s' command with a
possibly empty LHS) more consistent:

       pattern               GNU sed 3.x       GNU sed 4.x
        B                      xBx               xBx
        BC                     xBxCx             xBxCx
        BAC                    xBxxCx            xBxCx
        BAAC                   xBxxCx            xBxCx

* Bug fix: the // empty regular expressions now refers to the last
regular expression that was matched, rather than to the last
regular expression that was compiled.  This richer behavior seems
to be the correct one (albeit neither one is POSIXLY_CORRECT).

* Check for invalid backreferences in the RHS of the `s' command
(e.g. s/1234/\1/)

* Support for \[lLuUE] in the RHS of the `s' command like in Perl.

* New regular expression matcher

* Bug fix: if a file was redirected to be stdin, sed did not consume
it.  So
      (sed d; sed G) < TESTFILE

double-spaced TESTFILE, while the equivalent `useless use of cat'
      cat TESTFILE | (sed d; sed G)

printed nothing (which is the correct behavior).  A test for this
bug was added to the test suite.

* The documentation is now much better, with a few examples provided,
and a thorough description of regular expressions.  The manual often
refers to "GNU extensions", but if they are described here they are
specific to this version.

* Documented command-line option:
  -r, --regexp-extended
    Use extended regexps -- e.g. (abc+) instead of \(abc\+\)

* Added feature to the `w' command and to the `w' option of the `s'
command: if the file name is /dev/stderr, it means the standard
error (inspired by awk); and similarly for /dev/stdout.  This is
disabled if POSIXLY_CORRECT is set.

* Added `m' and `M' modifiers to `s' command for multi-line
matching (Perl-style); in addresses, only `M' works.

* Added `Q' command for `silent quit'; added ability to pass
an exit code from a sed script to the caller.

* Added `T' command for `branch if failed'.

* Added `v' command, which is a do-nothing intended to fail on
seds that do not support GNU sed 4.0's extensions.

----------------------------------------------------------------------------
Sed 3.02.80

* Started new version nomenclature for pre-3.03 releases.  (I'm being
pessimistic in assuming that .90 won't give me enough breathing room.)

* Bug fixes: the regncomp()/regnexec() interfaces proved to be inadequate to
properly handle expressions such as "s/\</#/g".  Re-abstracted the regex
code in the sed/ tree, and now use the re_search_2() interface to the GNU
regex routines.  This change also fixed a bug where /./ did not match the
NUL character.  Had the glibc folk fix a bug in lib/regex.c where
's/0*\([0-9][0-9]\)/X\1X/' failed to match on input "002".

* Added new command-line options:
  -u, --unbuffered
    Do not attempt to read-ahead more than required; do not buffer stdout.
  -l N, --line-length=N
    Specify the desired line-wrap length for the `l' command.
    A length of "0" means "never wrap".

* New internationalization translations added: fr ru de it el sk pt_BR sv
(plus nl from 3.02a).

* The s/// command now understands the following escapes
(in both halves):
        \a      an "alert" (BEL)
        \f      a form-feed
        \n      a newline
        \r      a carriage-return
        \t      a horizontal tab
        \v      a vertical tab
        \oNNN   a character with the octal value NNN
        \dNNN   a character with the decimal value NNN
        \xNN    a character with the hexadecimal value NN
This behavior is disabled if POSIXLY_CORRECT is set, at least for the
time being (until I can be convinced that this behavior does not violate
the POSIX standard).  (Incidentally, \b (backspace) was omitted because
of the conflict with the existing "word boundary" meaning. \ooo octal
format was omitted because of the conflict with backreference syntax.)

* If POSIXLY_CORRECT is set, the empty RE // now is the null match
instead of "repeat the last REmatch".  As far as I can tell
this behavior is mandated by POSIX, but it would break too many
legacy sed scripts to blithely change GNU sed's default behavior.

----------------------------------------------------------------------------
Sed 3.02a

* Added internationalization support, and an initial (already out of date)
set of Dutch message translations (both provided by Erick Branderhorst).

* Added support for scripts like:
 sed -e 1ifoo -e '$abar'
(note no need for \ <newline> after a, i, and c commands).
Also, conditionally (on NO_INPUT_INDENT) added
experimental support for skipping leading whitespace on
each {a,i,c} input line.

* Added addressing of the form:
 /foo/,+5 p (print from foo to 5th line following)
 /foo/,~5 p (print from foo to next line whose line number is a multiple of 5)
The first address of these can be any of the previously existing
addressing types; the +N and ~N forms are only allowed as the
second address of a range.

* Added support for pseudo-address "0" as the first address in an
address-range, simplifying scripts which happen to match the end
address on the first line of input.  For example, a script
which deletes all lines from the beginning of the file to the
first line which contains "foo" is now simply "sed 0,/foo/d",
whereas before one had to go through contortions to deal with
the possibility that "foo" might appear on the first line of
the input.

* Made NUL characters in regexps work "correctly" --- i.e., a NUL
in a RE matches a NUL; it does not prematurely terminate the RE.
(This only works in -f scripts, as the POSIX.1 exec*() interface
only passes NUL-terminated strings, and so sed will only be able
to see up to the first NUL in any -e scriptlet.)

* Wherever a `;' is accepted as a command terminator, also allow a `}'
or a `#' to appear.  (This allows for less cluttered-looking scripts.)

* Lots of internal changes that are only relevant to source junkies
and development testing.  Some of which might cause imperceptible
performance improvements.

----------------------------------------------------------------------------
Sed 3.02

* Fixed a bug in the parsing of character classes (e.g., /[[:space:]]/).
Corrected an omission in djgpp/Makefile.am and an improper dependency
in testsuite/Makefile.am.

----------------------------------------------------------------------------
Sed 3.01

* This version of sed mainly contains bug fixes and portability
enhancements, plus performance enhancements related to sed's handling
of input files.  Due to excess performance penalties, I have reverted
(relative to 3.00) to using regex.c instead of the rx package for
regular expression handling, at the expense of losing true POSIX.2
BRE compatibility.  However, performance related to regular expression
handling *still* needs a fair bit of work.

* One new feature has been added: regular expressions may be followed
with an "I" directive ("i" was taken [the "i"nsert command]) to
indicate that the regexp should be matched in a case-insensitive
manner.  Also of note are a new organization to the source code,
new documentation, and a new maintainer.

----------------------------------------------------------------------------
Sed 3.0

* This version of sed passes the new test-suite donated by
Jason Molenda.

* Overall performance has been improved in the following sense: Sed 3.0
is often slightly slower than sed 2.05.  On a few scripts, though, sed
2.05 was so slow as to be nearly useless or to use up unreasonable
amounts of memory.  These problems have been fixed and in such cases,
sed 3.0 should have acceptable performance.