Errors indexing past the end of a List

[p6rt]

Migrated from rt.perl.org#131699 (status was 'open')

Searchable as RT131699$

[p6rt]

From @briandfoy

Accessing a List element beyond the end of the List returns Nil,
although accessing an element before the beginning returns an out of
bounds failure. I think there's two things that can be better here
since we know the size of the List.

  my $list = < a b c >;
  put "I have a {$list.^name}";

First, in the "before" case, we have more information than the error
message lets on. The index should be from 0 to 2​:

  {
  my $i = -1;
  $list[$i]; # Index out of range. Is​: -1, should be in 0..^Inf
  }

But this requires the change I think is more helpful. Since the List
size won't change, we can have the same out-of-bounds error on
accesses past the end. At the moment it's no warning​:

  {
  my $i = $list.end + 1;
  $list[$i]; # No warning
  }

This would then be the error for assigning into a position beyond the
end. The existing error doesn't say what went wrong even though Perl 6
has enough information to figure that out​:

  {
  my $i = $list.end + 1;
  $list[$i] = 5; # Cannot modify an immutable Nil
  }

[p6rt]

From @AlexDaniel

Hm. Wouldn't that make behavior of Lists and Arrays different?

On 2017-07-04 05​:29​:20, comdog wrote​:

Accessing a List element beyond the end of the List returns Nil,
although accessing an element before the beginning returns an out of
bounds failure. I think there's two things that can be better here
since we know the size of the List.

my $list = < a b c >;
put "I have a {$list.^name}";

First, in the "before" case, we have more information than the error
message lets on. The index should be from 0 to 2​:

{
my $i = -1;
$list[$i]; # Index out of range. Is​: -1, should be in 0..^Inf
}

But this requires the change I think is more helpful. Since the List
size won't change, we can have the same out-of-bounds error on
accesses past the end. At the moment it's no warning​:

{
my $i = $list.end + 1;
$list[$i]; # No warning
}

This would then be the error for assigning into a position beyond the
end. The existing error doesn't say what went wrong even though Perl 6
has enough information to figure that out​:

{
my $i = $list.end + 1;
$list[$i] = 5; # Cannot modify an immutable Nil
}

[p6rt]

The RT System itself - Status changed from 'new' to 'open'

[p6rt]

From @lizmat

On 4 Jul 2017, at 16​:05, Aleks-Daniel Jakimenko-Aleksejev via RT <perl6-bugs-followup@​perl.org> wrote​:
On 2017-07-04 05​:29​:20, comdog wrote​:

Accessing a List element beyond the end of the List returns Nil,
although accessing an element before the beginning returns an out of
bounds failure. I think there's two things that can be better here
since we know the size of the List.

my $list = < a b c >;
put "I have a {$list.^name}";

First, in the "before" case, we have more information than the error
message lets on. The index should be from 0 to 2​:

{
my $i = -1;
$list[$i]; # Index out of range. Is​: -1, should be in 0..^Inf
}

But this requires the change I think is more helpful. Since the List
size won't change, we can have the same out-of-bounds error on
accesses past the end. At the moment it's no warning​:

{
my $i = $list.end + 1;
$list[$i]; # No warning
}

This would then be the error for assigning into a position beyond the
end. The existing error doesn't say what went wrong even though Perl 6
has enough information to figure that out​:

{
my $i = $list.end + 1;
$list[$i] = 5; # Cannot modify an immutable Nil
}
Hm. Wouldn't that make behavior of Lists and Arrays different?

No, because Lists are supposed to be immutable wrt to the number of elements.

That said, a List may not always be completely reified already. So logically, a List may have 100 elements, it could well be that only 42 of these elements exist already. Which means that the underlying NQP array, which *is* mutable, only has 42 elements. But it cannot know offhand whether those are all possible elements, as that depends on the iterator that is being used to fill the NQP array.

A complicating factor is that this is a very hot code path, so any changes there could affect general performance significantly. My initial tests to generate a Failure on out of bounds value immediately results in 2 internal errors trying to generate backtrace :-(

Anyways, I agree with brian’s feelings on this. The challenge is now to make it so without making everything significantly slower.

FWIW, the code in question lives around line 480 in List.pm.

Liz

[p6rt]

From @lizmat

On 4 Jul 2017, at 16​:19, Elizabeth Mattijsen <liz@​dijkmat.nl> wrote​:
That said, a List may not always be completely reified already. So logically, a List may have 100 elements, it could well be that only 42 of these elements exist already. Which means that the underlying NQP array, which *is* mutable, only has 42 elements. But it cannot know offhand whether those are all possible elements, as that depends on the iterator that is being used to fill the NQP array.

A complicating factor is that this is a very hot code path, so any changes there could affect general performance significantly. My initial tests to generate a Failure on out of bounds value immediately results in 2 internal errors trying to generate backtrace :-(

Anyways, I agree with brian’s feelings on this. The challenge is now to make it so without making everything significantly slower.

FWIW, the code in question lives around line 480 in List.pm.

An example of a List that will never be fully reified​:

  $ 6 'my $l = ^Inf .list; dd $l.^name; dd $l[10]'
  "List"
  10

Liz

[p6rt]

From @zoffixznet

On Tue, 04 Jul 2017 07​:20​:50 -0700, elizabeth wrote​:

Hm. Wouldn't that make behavior of Lists and Arrays different?

No, because Lists are supposed to be immutable wrt to the number of
elements.

Yes, but that doesn't mean the user of the list necessarily knows or has to know how many elements the list has, to look up an element by index, without things exploding.

generate a Failure on out of bounds value

IMO that's the wrong approach and the current behaviour of returning Nil is most desirable. Consider this code​:

  sub do-stuff (@​args) {
  say @​args[0] + 1;
  }

  do-stuff [];
  do-stuff ();

If you Fatalise past-end .AT-POS on Lists that sub will now work fine with Arrays, but crash with Lists. So as a consumer of `@​args` Iterable, I'm now faced with two potential behaviours and have to account for Exceptions.

Currently, the code will also produce a warning, as do many things that consume a Nil where Nil doesn't really belong. IMO that's a good-enough warning for the inadvertent out-of-bounds access.

By adding the Failure, we also end up with another inconsistency that `my ($a, $b) = @​list` would work and assign Nil to `$b` when `@​list` has just one element, but `my ($a, $b) = @​list[0,1]` would assign a Failure to it. Lastly, we'd have `@​list[0,1]` return a list with 1 value and 1 Failure in it (call .elems on it and the Failure's silently gone).

So, IMO, yes, we could change "0..^Inf" to tell number of actual elements, but that error is fundamentally different from trying to access past List's end, because you can NEVER access element at index -1, yet you can access index N, and we should leave the "return Nil" behaviour as is and not deviate from Array behaviour.

[AlexDaniel]

TL;DR What would be the best behavior for these?

my $list = < a b c >; 
say $list[- 1]; # note that it needs to be “- 1” and not “-1”
say $list[42];