Discussion:
[Mojolicious] Mojo::UserAgent URL with caret '^' not encoded
Sylvain Thibault
2018-10-27 23:38:43 UTC
Permalink
Given URL https://somehost.com.com/streamer/1.0?s=^GSPC

When doing a GET, Mojo::UserAgent encodes the caret '^' as %5E
The server returns NOT FOUND.

So this transaction from Mojo::UserAgent returns NOT FOUND:

GET /streamer/1.0?s=%5EGSPC&k=l86,l84,p20 HTTP/1.1
Host: somehost.com
Accept: */*
User-Agent: Mojolicious (Perl)
Content-Length: 0

This transaction using curl with the caret not encoded returns the desired
output:

GET /streamer/1.0?s=^GSPC&k=l86,l84,p20 HTTP/1.1
Host: s <http://streamerapi.finance.yahoo.com/>omehost.com
User-Agent: curl/7.61.1
Accept: */*

How does one send a caret '^' in a URL without encoding it to %5E ?

Using a symbol without a caret works great in the Mojo::UserAgent version
of the code.

Thanks,

Sylvain Thibault
--
You received this message because you are subscribed to the Google Groups "Mojolicious" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mojolicious+***@googlegroups.com.
To post to this group, send email to ***@googlegroups.com.
Visit this group at https://groups.google.com/group/mojolicious.
For more options, visit https://groups.google.com/d/optout.
Stefan Adams
2018-10-28 16:32:36 UTC
Permalink
Post by Sylvain Thibault
How does one send a caret '^' in a URL without encoding it to %5E ?
Using a symbol without a caret works great in the Mojo::UserAgent version
of the code.
Being that we're talking about a query parameter, it seems like we're not
dealing with a routing problem. Can you see from the log if your action
handler is being processed? Is so, add `warn $c->param('s')` right after
you set $c to the first parameter and then watch STDERR for what's
contained in the 's' parameter. Do you have any code in your action
handler that returns a 404, such as $c->reply->not_found?
--
You received this message because you are subscribed to the Google Groups "Mojolicious" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mojolicious+***@googlegroups.com.
To post to this group, send email to ***@googlegroups.com.
Visit this group at https://groups.google.com/group/mojolicious.
For more options, visit https://groups.google.com/d/optout.
Sylvain Thibault
2018-10-28 16:49:15 UTC
Permalink
Thank you for your response Stefan.
Here is the complete code. Symbol SPY works, symbol ^GSPC returns not found
.

This curl command with the ^GSPC symbol works:

curl -s -o - -N -v
'https://streamerapi.finance.yahoo.com/streamer/1.0?s=^GSPC&k=l86,l84,p20&callback=parent.yfs_u1f&mktmcb=parent.yfs_mktmcb&gencallback=parent.yfs_gencb&mu=1&lang=en-US&region=US&localize=0'


#! /usr/bin/env perl
use Mojo::UserAgent;
use feature qw(say);

$| = 1;

# Accept responses of indefinite size
my $ua = Mojo::UserAgent->new(max_response_size => 0);
$ua->inactivity_timeout(0);

my $url_raw
# Symbol ^GSPC returns NOT FOUND
# =
'https://streamerapi.finance.yahoo.com/streamer/1.0?s=^GSPC&k=l86,l84,p20&callback=parent.yfs_u1f&mktmcb=parent.yfs_mktmcb&gencallback=parent.yfs_gencb&mu=1&lang=en-US&region=US&localize=0';
=
'https://streamerapi.finance.yahoo.com/streamer/1.0?s=SPY&k=l86,l84,p20&callback=parent.yfs_u1f&mktmcb=parent.yfs_mktmcb&gencallback=parent.yfs_gencb&mu=1&lang=en-US&region=US&localize=0'
;


# Build a normal transaction
my $tx = $ua->build_tx(
GET => $url_raw,
=> {Accept => '*/*'}
);

# Remove caret encoding... results is not found... ???
say $tx->req->to_string;

# Replace "read" events to disable default content parser
$tx->res->content->unsubscribe('read')->on(
read => sub {
my ($content, $bytes) = @_;
say "Streaming: $bytes";
}
);

# Process transaction
$tx = $ua->start($tx);


say "done";
Post by Sylvain Thibault
Given URL https://somehost.com.com/streamer/1.0?s=^GSPC
When doing a GET, Mojo::UserAgent encodes the caret '^' as %5E
The server returns NOT FOUND.
GET /streamer/1.0?s=%5EGSPC&k=l86,l84,p20 HTTP/1.1
Host: somehost.com
Accept: */*
User-Agent: Mojolicious (Perl)
Content-Length: 0
This transaction using curl with the caret not encoded returns the desired
GET /streamer/1.0?s=^GSPC&k=l86,l84,p20 HTTP/1.1
Host: s <http://streamerapi.finance.yahoo.com/>omehost.com
User-Agent: curl/7.61.1
Accept: */*
How does one send a caret '^' in a URL without encoding it to %5E ?
Using a symbol without a caret works great in the Mojo::UserAgent version
of the code.
Thanks,
Sylvain Thibault
--
You received this message because you are subscribed to the Google Groups "Mojolicious" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mojolicious+***@googlegroups.com.
To post to this group, send email to ***@googlegroups.com.
Visit this group at https://groups.google.com/group/mojolicious.
For more options, visit https://groups.google.com/d/optout.
Stefan Adams
2018-10-28 18:46:22 UTC
Permalink
I see, you are building a client app, not a server app -- I missed that
detail.

And, I see that you want to pass the ^ *without* escaping it.

Your URL is getting parsed by Mojo::URL, whose parameters are getting
parsed by Mojo::Parameters, and when you get the parameters as a string
<https://mojolicious.org/perldoc/Mojo/Parameters#to_string>, url_escape
<https://mojolicious.org/perldoc/Mojo/Util#url_escape> is being applied:

Percent encode unsafe characters in string as described in RFC 3986
<http://tools.ietf.org/html/rfc3986>, the pattern used defaults to
^A-Za-z0-9\-._~.


Reading the code for to_string in Mojo::Parameters, I see no way around
this. It would seem that Yahoo Finance is not RFC 3986 compliant. The
question that needs to be asked then is, "Is it possible for Mojo::URL to
customize the characters that are considered unsafe?" I don't know super
advanced Perl like this, but maybe it's possible to override a function and
make Mojo::Util::url_escape not actually escape the string?

There might be a more advanced way to build your transaction, one that
doesn't operate on a Mojo::URL object which will ultimately apply
url_escape without question -- but I wouldn't know it.

Anyway, from what I can see, there's not a solution to your problem. :(
Post by Sylvain Thibault
Thank you for your response Stefan.
Here is the complete code. Symbol SPY works, symbol ^GSPC returns not
found.
curl -s -o - -N -v 'https://streamerapi.finance.yahoo.com/streamer/1.0?s=
^GSPC&k=l86,l84,p20&callback=parent.yfs_u1f&mktmcb=parent.yfs_mktmcb&gencallback=parent.yfs_gencb&mu=1&lang=en-US&region=US&localize=0'
#! /usr/bin/env perl
use Mojo::UserAgent;
use feature qw(say);
$| = 1;
# Accept responses of indefinite size
my $ua = Mojo::UserAgent->new(max_response_size => 0);
$ua->inactivity_timeout(0);
my $url_raw
# Symbol ^GSPC returns NOT FOUND
# = 'https://streamerapi.finance.yahoo.com/streamer/1.0?s=
^GSPC&k=l86,l84,p20&callback=parent.yfs_u1f&mktmcb=parent.yfs_mktmcb&gencallback=parent.yfs_gencb&mu=1&lang=en-US&region=US&localize=0';
= '
https://streamerapi.finance.yahoo.com/streamer/1.0?s=SPY&k=l86,l84,p20&callback=parent.yfs_u1f&mktmcb=parent.yfs_mktmcb&gencallback=parent.yfs_gencb&mu=1&lang=en-US&region=US&localize=0
';
# Build a normal transaction
my $tx = $ua->build_tx(
GET => $url_raw,
=> {Accept => '*/*'}
);
# Remove caret encoding... results is not found... ???
say $tx->req->to_string;
# Replace "read" events to disable default content parser
$tx->res->content->unsubscribe('read')->on(
read => sub {
say "Streaming: $bytes";
}
);
# Process transaction
$tx = $ua->start($tx);
say "done";
Post by Sylvain Thibault
Given URL https://somehost.com.com/streamer/1.0?s=^GSPC
When doing a GET, Mojo::UserAgent encodes the caret '^' as %5E
The server returns NOT FOUND.
GET /streamer/1.0?s=%5EGSPC&k=l86,l84,p20 HTTP/1.1
Host: somehost.com
Accept: */*
User-Agent: Mojolicious (Perl)
Content-Length: 0
This transaction using curl with the caret not encoded returns the
GET /streamer/1.0?s=^GSPC&k=l86,l84,p20 HTTP/1.1
Host: s <http://streamerapi.finance.yahoo.com/>omehost.com
User-Agent: curl/7.61.1
Accept: */*
How does one send a caret '^' in a URL without encoding it to %5E ?
Using a symbol without a caret works great in the Mojo::UserAgent version
of the code.
Thanks,
Sylvain Thibault
--
You received this message because you are subscribed to the Google Groups
"Mojolicious" group.
To unsubscribe from this group and stop receiving emails from it, send an
Visit this group at https://groups.google.com/group/mojolicious.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "Mojolicious" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mojolicious+***@googlegroups.com.
To post to this group, send email to ***@googlegroups.com.
Visit this group at https://groups.google.com/group/mojolicious.
For more options, visit https://groups.google.com/d/optout.
Sylvain Thibault
2018-10-28 19:22:40 UTC
Permalink
Thanks for that.
You would think Yahoo would be RFC 3986 compliant but their services have
been on the decline for a while.

I don't know how to override Mojo::Util url_escape either.
Post by Stefan Adams
I see, you are building a client app, not a server app -- I missed that
detail.
And, I see that you want to pass the ^ *without* escaping it.
Your URL is getting parsed by Mojo::URL, whose parameters are getting
parsed by Mojo::Parameters, and when you get the parameters as a string
<https://mojolicious.org/perldoc/Mojo/Parameters#to_string>, url_escape
Percent encode unsafe characters in string as described in RFC 3986
<http://tools.ietf.org/html/rfc3986>, the pattern used defaults to
^A-Za-z0-9\-._~.
Reading the code for to_string in Mojo::Parameters, I see no way around
this. It would seem that Yahoo Finance is not RFC 3986 compliant. The
question that needs to be asked then is, "Is it possible for Mojo::URL to
customize the characters that are considered unsafe?" I don't know super
advanced Perl like this, but maybe it's possible to override a function and
make Mojo::Util::url_escape not actually escape the string?
There might be a more advanced way to build your transaction, one that
doesn't operate on a Mojo::URL object which will ultimately apply
url_escape without question -- but I wouldn't know it.
Anyway, from what I can see, there's not a solution to your problem. :(
Post by Sylvain Thibault
Thank you for your response Stefan.
Here is the complete code. Symbol SPY works, symbol ^GSPC returns not
found.
curl -s -o - -N -v 'https://streamerapi.finance.yahoo.com/streamer/1.0?s=
^GSPC&k=l86,l84,p20&callback=parent.yfs_u1f&mktmcb=parent.yfs_mktmcb&gencallback=parent.yfs_gencb&mu=1&lang=en-US&region=US&localize=0'
#! /usr/bin/env perl
use Mojo::UserAgent;
use feature qw(say);
$| = 1;
# Accept responses of indefinite size
my $ua = Mojo::UserAgent->new(max_response_size => 0);
$ua->inactivity_timeout(0);
my $url_raw
# Symbol ^GSPC returns NOT FOUND
# = 'https://streamerapi.finance.yahoo.com/streamer/1.0?s=
^GSPC&k=l86,l84,p20&callback=parent.yfs_u1f&mktmcb=parent.yfs_mktmcb&gencallback=parent.yfs_gencb&mu=1&lang=en-US&region=US&localize=0';
= '
https://streamerapi.finance.yahoo.com/streamer/1.0?s=SPY&k=l86,l84,p20&callback=parent.yfs_u1f&mktmcb=parent.yfs_mktmcb&gencallback=parent.yfs_gencb&mu=1&lang=en-US&region=US&localize=0
';
# Build a normal transaction
my $tx = $ua->build_tx(
GET => $url_raw,
=> {Accept => '*/*'}
);
# Remove caret encoding... results is not found... ???
say $tx->req->to_string;
# Replace "read" events to disable default content parser
$tx->res->content->unsubscribe('read')->on(
read => sub {
say "Streaming: $bytes";
}
);
# Process transaction
$tx = $ua->start($tx);
say "done";
Post by Sylvain Thibault
Given URL https://somehost.com.com/streamer/1.0?s=^GSPC
When doing a GET, Mojo::UserAgent encodes the caret '^' as %5E
The server returns NOT FOUND.
GET /streamer/1.0?s=%5EGSPC&k=l86,l84,p20 HTTP/1.1
Host: somehost.com
Accept: */*
User-Agent: Mojolicious (Perl)
Content-Length: 0
This transaction using curl with the caret not encoded returns the
GET /streamer/1.0?s=^GSPC&k=l86,l84,p20 HTTP/1.1
Host: s <http://streamerapi.finance.yahoo.com/>omehost.com
User-Agent: curl/7.61.1
Accept: */*
How does one send a caret '^' in a URL without encoding it to %5E ?
Using a symbol without a caret works great in the Mojo::UserAgent
version of the code.
Thanks,
Sylvain Thibault
--
You received this message because you are subscribed to the Google Groups
"Mojolicious" group.
To unsubscribe from this group and stop receiving emails from it, send an
<javascript:>.
Visit this group at https://groups.google.com/group/mojolicious.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "Mojolicious" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mojolicious+***@googlegroups.com.
To post to this group, send email to ***@googlegroups.com.
Visit this group at https://groups.google.com/group/mojolicious.
For more options, visit https://groups.google.com/d/optout.
Heiko Jansen
2018-10-31 11:09:43 UTC
Permalink
I do not know of a non-ugly way either but if you are okay with a hacky and
possibly fragile solution you might get away with adding:

$ua->on(start => sub {
my ($ua, $tx) = @_;
$tx->req->{'start_buffer'} =~ s/([&?]s=)%5E/$1^/o;
});

Poking into internals usually means one day you'll get what you asked for,
but if there is no other way...
--
You received this message because you are subscribed to the Google Groups "Mojolicious" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mojolicious+***@googlegroups.com.
To post to this group, send email to ***@googlegroups.com.
Visit this group at https://groups.google.com/group/mojolicious.
For more options, visit https://groups.google.com/d/optout.
Sylvain Thibault
2018-10-31 15:44:15 UTC
Permalink
Thank you Heiko,
This works for me.
This case is definitely an outlier.
Post by Heiko Jansen
I do not know of a non-ugly way either but if you are okay with a hacky
$ua->on(start => sub {
$tx->req->{'start_buffer'} =~ s/([&?]s=)%5E/$1^/o;
});
Poking into internals usually means one day you'll get what you asked for,
but if there is no other way...
--
You received this message because you are subscribed to the Google Groups "Mojolicious" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mojolicious+***@googlegroups.com.
To post to this group, send email to ***@googlegroups.com.
Visit this group at https://groups.google.com/group/mojolicious.
For more options, visit https://groups.google.com/d/optout.
sri
2018-10-31 15:48:38 UTC
Permalink
Post by Heiko Jansen
I do not know of a non-ugly way either but if you are okay with a hacky
$ua->on(start => sub {
$tx->req->{'start_buffer'} =~ s/([&?]s=)%5E/$1^/o;
});
Poking into internals usually means one day you'll get what you asked for,
but if there is no other way...
Yes, that is a very dangerous hack, a more sane approach would be to use a
Role::Tiny role modifying
Mojo::Parameters::to_string
($tx->req->url->query->with_roles('+Unescaped')) or something similar.

--
sebastian
--
You received this message because you are subscribed to the Google Groups "Mojolicious" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mojolicious+***@googlegroups.com.
To post to this group, send email to ***@googlegroups.com.
Visit this group at https://groups.google.com/group/mojolicious.
For more options, visit https://groups.google.com/d/optout.
Stefan Adams
2018-10-31 16:10:16 UTC
Permalink
Post by Heiko Jansen
Poking into internals usually means one day you'll get what you asked for,
Post by Heiko Jansen
but if there is no other way...
Yes, that is a very dangerous hack, a more sane approach would be to use a
Role::Tiny role modifying
Mojo::Parameters::to_string
($tx->req->url->query->with_roles('+Unescaped')) or something similar.
Yes! I love this strategy! Existing packages like Mojo::Collection and
Mojo::File already have with_roles available, but Mojo::Parameters does
not. Does it make sense to add that, or is it really that simple to just
add it independently and the reason why with_roles is available to these
other packages is just for convenience due to the overwhelming likelihood
that they will be used more frequently?
--
You received this message because you are subscribed to the Google Groups "Mojolicious" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mojolicious+***@googlegroups.com.
To post to this group, send email to ***@googlegroups.com.
Visit this group at https://groups.google.com/group/mojolicious.
For more options, visit https://groups.google.com/d/optout.
Dan Book
2018-10-31 16:14:24 UTC
Permalink
It does have with_roles, it comes from Mojo::Base.

-Dan
Post by Stefan Adams
Post by Heiko Jansen
Poking into internals usually means one day you'll get what you asked
for, but if there is no other way...
Yes, that is a very dangerous hack, a more sane approach would be to use
a Role::Tiny role modifying
Mojo::Parameters::to_string
($tx->req->url->query->with_roles('+Unescaped')) or something similar.
Yes! I love this strategy! Existing packages like Mojo::Collection and
Mojo::File already have with_roles available, but Mojo::Parameters does
not. Does it make sense to add that, or is it really that simple to just
add it independently and the reason why with_roles is available to these
other packages is just for convenience due to the overwhelming likelihood
that they will be used more frequently?
--
You received this message because you are subscribed to the Google Groups
"Mojolicious" group.
To unsubscribe from this group and stop receiving emails from it, send an
Visit this group at https://groups.google.com/group/mojolicious.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "Mojolicious" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mojolicious+***@googlegroups.com.
To post to this group, send email to ***@googlegroups.com.
Visit this group at https://groups.google.com/group/mojolicious.
For more options, visit https://groups.google.com/d/optout.
Stefan Adams
2018-10-31 16:42:37 UTC
Permalink
Post by Dan Book
It does have with_roles, it comes from Mojo::Base.
Hmm!! I wondered that!!! But then why isn't it documented like it is for
Mojo::Collection and Mojo::File? What difference is there?

Gosh, knowing this now opens up a lot of doors for me!!!
--
You received this message because you are subscribed to the Google Groups "Mojolicious" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mojolicious+***@googlegroups.com.
To post to this group, send email to ***@googlegroups.com.
Visit this group at https://groups.google.com/group/mojolicious.
For more options, visit https://groups.google.com/d/optout.
Dan Book
2018-10-31 17:18:01 UTC
Permalink
Mojo::Collection and Mojo::File are not hash-based objects so they cannot
use Mojo::Base, thus they implement with_roles themselves. Mojo::Base based
objects all say "inherits all methods from Mojo::Base" or a subclass of
that.

-Dan
Post by Stefan Adams
Post by Dan Book
It does have with_roles, it comes from Mojo::Base.
Hmm!! I wondered that!!! But then why isn't it documented like it is for
Mojo::Collection and Mojo::File? What difference is there?
Gosh, knowing this now opens up a lot of doors for me!!!
--
You received this message because you are subscribed to the Google Groups
"Mojolicious" group.
To unsubscribe from this group and stop receiving emails from it, send an
Visit this group at https://groups.google.com/group/mojolicious.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "Mojolicious" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mojolicious+***@googlegroups.com.
To post to this group, send email to ***@googlegroups.com.
Visit this group at https://groups.google.com/group/mojolicious.
For more options, visit https://groups.google.com/d/optout.
Stefan Adams
2018-10-31 21:26:48 UTC
Permalink
Post by Dan Book
Mojo::Collection and Mojo::File are not hash-based objects so they cannot
use Mojo::Base, thus they implement with_roles themselves. Mojo::Base based
objects all say "inherits all methods from Mojo::Base" or a subclass of
that.
This is really helpful! Thank you for clearing that up!
--
You received this message because you are subscribed to the Google Groups "Mojolicious" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mojolicious+***@googlegroups.com.
To post to this group, send email to ***@googlegroups.com.
Visit this group at https://groups.google.com/group/mojolicious.
For more options, visit https://groups.google.com/d/optout.
Sylvain Thibault
2018-11-01 00:09:41 UTC
Permalink
Can somebody describe with a little more detail how one would go about
using a role to modify Mojo::Parameters::to_string
and using it with something like
$tx->req->url->query->with_roles('+Unescaped') ?

I am looking at Joel Berger's blog
post: https://mojolicious.io/blog/2017/12/13/day-13-more-about-roles/
but not quite getting there.
Post by Sylvain Thibault
Given URL https://somehost.com.com/streamer/1.0?s=^GSPC
When doing a GET, Mojo::UserAgent encodes the caret '^' as %5E
The server returns NOT FOUND.
GET /streamer/1.0?s=%5EGSPC&k=l86,l84,p20 HTTP/1.1
Host: somehost.com
Accept: */*
User-Agent: Mojolicious (Perl)
Content-Length: 0
This transaction using curl with the caret not encoded returns the desired
GET /streamer/1.0?s=^GSPC&k=l86,l84,p20 HTTP/1.1
Host: s <http://streamerapi.finance.yahoo.com/>omehost.com
User-Agent: curl/7.61.1
Accept: */*
How does one send a caret '^' in a URL without encoding it to %5E ?
Using a symbol without a caret works great in the Mojo::UserAgent version
of the code.
Thanks,
Sylvain Thibault
--
You received this message because you are subscribed to the Google Groups "Mojolicious" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mojolicious+***@googlegroups.com.
To post to this group, send email to ***@googlegroups.com.
Visit this group at https://groups.google.com/group/mojolicious.
For more options, visit https://groups.google.com/d/optout.
Sylvain Thibault
2018-11-01 02:27:38 UTC
Permalink
As suggested, here is a working solution using a Role::Tiny role to modify
Mojo::Parameters::to_string so that it does not escape the caret '^'
character.
Feel free to comment/criticize.

Thank you all for you valuable input, this was enlightening.

use Mojo::UserAgent;
use FindBin qw($Bin);
use feature qw(say);
use lib $Bin;

$| = 1;

# Accept responses of indefinite size
my $ua = Mojo::UserAgent->new(max_response_size => 0);
$ua->inactivity_timeout(0);

my $url_raw
=
'https://streamerapi.finance.yahoo.com/streamer/1.0?s=^GSPC&k=l86,l84,p20&callback=parent.yfs_u1f&mktmcb=parent.yfs_mktmcb&gencallback=parent.yfs_gencb&mu=1&lang=en-US&region=US&localize=0'
;

# Build a normal transaction
my $tx = $ua->build_tx(GET => $url_raw, => {Accept => '*/*'});

# Bypass caret '^' encoding as Yahoo finance servers are not RFC 3986
compliant yet...
$tx->req->url->query->with_roles('+Unescaped');

say $tx->req->to_string;

# Replace "read" events to disable default content parser
$tx->res->content->unsubscribe('read')->on(
read => sub {
my ($content, $bytes) = @_;
say "Streaming: $bytes";
}
);

# Process transaction
$tx = $ua->start($tx);

say "done";


And the file ./Mojo/Parameters/Role/Unescaped.pm to modify/override
Mojo::Parameters::to_string

package Mojo::Parameters::Role::Unescaped;


use Mojo::Base -role;


use Mojo::Util qw(decode encode url_escape url_unescape);


has charset => 'UTF-8';


sub to_string {
my $self = shift;


# String (RFC 3986)
my $charset = $self->charset;
if (defined(my $str = $self->{string})) {
$str = encode $charset, $str if $charset;
return url_escape $str, '^A-Za-z0-9\-._~%!$&\'()*+,;=:@/?i^';
}
}




1;
Post by Sylvain Thibault
Given URL https://somehost.com.com/streamer/1.0?s=^GSPC
When doing a GET, Mojo::UserAgent encodes the caret '^' as %5E
The server returns NOT FOUND.
GET /streamer/1.0?s=%5EGSPC&k=l86,l84,p20 HTTP/1.1
Host: somehost.com
Accept: */*
User-Agent: Mojolicious (Perl)
Content-Length: 0
This transaction using curl with the caret not encoded returns the desired
GET /streamer/1.0?s=^GSPC&k=l86,l84,p20 HTTP/1.1
Host: s <http://streamerapi.finance.yahoo.com/>omehost.com
User-Agent: curl/7.61.1
Accept: */*
How does one send a caret '^' in a URL without encoding it to %5E ?
Using a symbol without a caret works great in the Mojo::UserAgent version
of the code.
Thanks,
Sylvain Thibault
--
You received this message because you are subscribed to the Google Groups "Mojolicious" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mojolicious+***@googlegroups.com.
To post to this group, send email to ***@googlegroups.com.
Visit this group at https://groups.google.com/group/mojolicious.
For more options, visit https://groups.google.com/d/optout.
Sylvain Thibault
2018-11-01 02:34:27 UTC
Permalink
As suggested, here is a working solution using a Role::Tiny role to
modify/override Mojo::Parameters::to_string so that it does not escape the
caret '^' character.
Feel free to comment/criticize.

Thank you all for your valuable input, this was enlightening.

use Mojo::UserAgent;
use FindBin qw($Bin);
use feature qw(say);
use lib $Bin;

$| = 1;

# Accept responses of indefinite size
my $ua = Mojo::UserAgent->new(max_response_size => 0);
$ua->inactivity_timeout(0);

my $url_raw
= 'https://streamerapi.finance.yahoo.com/streamer/1.0?s=
^GSPC&k=l86,l84,p20&callback=parent.yfs_u1f&mktmcb=parent.yfs_mktmcb&gencallback=parent.yfs_gencb&mu=1&lang=en-US&region=US&localize=0'
;

# Build a normal transaction
my $tx = $ua->build_tx(GET => $url_raw, => {Accept => '*/*'});

# Bypass caret '^' encoding with a Role::Tiny role as Yahoo finance servers
are not RFC 3986 compliant yet...
$tx->req->url->query->with_roles('+Unescaped');

say $tx->req->to_string;

# Replace "read" events to disable default content parser
$tx->res->content->unsubscribe('read')->on(
read => sub {
my ($content, $bytes) = @_;
say "Streaming: $bytes";
}
);

# Process transaction
$tx = $ua->start($tx);

say "done";


And the file ./Mojo/Parameters/Role/Unescaped.pm to modify/override
Mojo::Parameters::to_string

package Mojo::Parameters::Role::Unescaped;

use Mojo::Base -role;

use Mojo::Util qw(decode encode url_escape url_unescape);

has charset => 'UTF-8';

# Override default Mojo::Parameters::to_string method
sub to_string {
my $self = shift;
# String (RFC 3986)
my $charset = $self->charset;
if (defined(my $str = $self->{string})) {
$str = encode $charset, $str if $charset;
# Do not encode caret '^' character
return url_escape $str, '^A-Za-z0-9\-._~%!$&\'()*+,;=:@/?i^';
}
}

1;
Post by Sylvain Thibault
Given URL https://somehost.com.com/streamer/1.0?s=^GSPC
When doing a GET, Mojo::UserAgent encodes the caret '^' as %5E
The server returns NOT FOUND.
GET /streamer/1.0?s=%5EGSPC&k=l86,l84,p20 HTTP/1.1
Host: somehost.com
Accept: */*
User-Agent: Mojolicious (Perl)
Content-Length: 0
This transaction using curl with the caret not encoded returns the desired
GET /streamer/1.0?s=^GSPC&k=l86,l84,p20 HTTP/1.1
Host: s <http://streamerapi.finance.yahoo.com/>omehost.com
User-Agent: curl/7.61.1
Accept: */*
How does one send a caret '^' in a URL without encoding it to %5E ?
Using a symbol without a caret works great in the Mojo::UserAgent version
of the code.
Thanks,
Sylvain Thibault
Given URL https://somehost.com.com/streamer/1.0?s=^GSPC
When doing a GET, Mojo::UserAgent encodes the caret '^' as %5E
The server returns NOT FOUND.
GET /streamer/1.0?s=%5EGSPC&k=l86,l84,p20 HTTP/1.1
Host: somehost.com
Accept: */*
User-Agent: Mojolicious (Perl)
Content-Length: 0
This transaction using curl with the caret not encoded returns the desired
GET /streamer/1.0?s=^GSPC&k=l86,l84,p20 HTTP/1.1
Host: s <http://streamerapi.finance.yahoo.com/>omehost.com
User-Agent: curl/7.61.1
Accept: */*
How does one send a caret '^' in a URL without encoding it to %5E ?
Using a symbol without a caret works great in the Mojo::UserAgent version
of the code.
Thanks,
Sylvain Thibault
--
You received this message because you are subscribed to the Google Groups "Mojolicious" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mojolicious+***@googlegroups.com.
To post to this group, send email to ***@googlegroups.com.
Visit this group at https://groups.google.com/group/mojolicious.
For more options, visit https://groups.google.com/d/optout.
Stefan Adams
2018-11-01 03:20:22 UTC
Permalink
Post by Sylvain Thibault
As suggested, here is a working solution using a Role::Tiny role to
modify/override Mojo::Parameters::to_string so that it does not escape the
caret '^' character.
Feel free to comment/criticize.
That looks terrific to me and pretty much exactly what I'd do!
Post by Sylvain Thibault
Thank you all for your valuable input, this was enlightening.
Heck yeah! I learned an important lesson myself! Thanks for your
question!
--
You received this message because you are subscribed to the Google Groups "Mojolicious" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mojolicious+***@googlegroups.com.
To post to this group, send email to ***@googlegroups.com.
Visit this group at https://groups.google.com/group/mojolicious.
For more options, visit https://groups.google.com/d/optout.
Loading...