[Scikit-learn-general] GSoC Project Proposal: Reinforcement Learning Module

Discussion:

Michał Koziarski

2016-03-02 19:21:43 UTC

Hello everyone,

As far as I can tell, except PyBrain (which doesn't seem to be actively
developed) there are no reinforcement learning libraries in Python. I was
wondering if community would be interested in using one and making it a
part of scikit-learn. Does it lie within the scope of the project?

Very raw idea is, as follows:
- to design common interface, similar to what is used in other parts of
scikit-sklearn;
- to implement established RL algorithms, reliant heavily on estimators
available in scikit-learn;
- and to prepare practical examples of what RL can be used for, to both
supplement documentation and encourage people not yet familiar with RL to
experiment with it in their own projects.

Once again, I would mostly like to know whether it event lies within the
scope of the project, or if it just won't be added because of project
philosophy. Other than that, I would obviously appreciate any feedback.

About me: I am a last master's CS student. My research interests involve
machine learning in general and reinforcement learning in particular; this
year I hope to start my PhD on the latter. My master's thesis revolves
around transfer learning in RL. I have experience with programming in
industry and on large projects.

Cheers,
MichaÅ

Andreas Mueller

2016-03-02 19:31:26 UTC

Permalink

Post by MichaÅ Koziarski
As far as I can tell, except PyBrain (which doesn't seem to be
actively developed) there are no reinforcement learning libraries in
Python. I was wondering if community would be interested in using one
and making it a part of scikit-learn. Does it lie within the scope of
the project?

It's not (and we should maybe make that more explicit in the FAQ).

Michał Koziarski

2016-03-02 19:39:19 UTC

Permalink

I see. Thank you for quick answer.

Post by Andreas Mueller

It's not (and we should maybe make that more explicit in the FAQ).
------------------------------------------------------------------------------
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140
_______________________________________________
Scikit-learn-general mailing list
https://lists.sourceforge.net/lists/listinfo/scikit-learn-general

Jacob Schreiber

2016-03-02 19:34:17 UTC

Permalink

Reinforcement learning is an exciting field of machine learning, and you're
right that it seems underrepresented in Python. However, I don't think that
it falls within the strict scope of the scikit-learn API.

Post by MichaÅ Koziarski
Hello everyone,
As far as I can tell, except PyBrain (which doesn't seem to be actively
developed) there are no reinforcement learning libraries in Python. I was
wondering if community would be interested in using one and making it a
part of scikit-learn. Does it lie within the scope of the project?
- to design common interface, similar to what is used in other parts of
scikit-sklearn;
- to implement established RL algorithms, reliant heavily on estimators
available in scikit-learn;
- and to prepare practical examples of what RL can be used for, to both
supplement documentation and encourage people not yet familiar with RL to
experiment with it in their own projects.
Once again, I would mostly like to know whether it event lies within the
scope of the project, or if it just won't be added because of project
philosophy. Other than that, I would obviously appreciate any feedback.
About me: I am a last master's CS student. My research interests involve
machine learning in general and reinforcement learning in particular; this
year I hope to start my PhD on the latter. My master's thesis revolves
around transfer learning in RL. I have experience with programming in
industry and on large projects.
Cheers,
MichaÅ
------------------------------------------------------------------------------
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140
_______________________________________________
Scikit-learn-general mailing list
https://lists.sourceforge.net/lists/listinfo/scikit-learn-general

Gael Varoquaux

2016-03-02 20:06:39 UTC

Permalink

Post by Jacob Schreiber
Reinforcement learning is an exciting field of machine learning, and you're
right that it seems underrepresented in Python. However, I don't think that it
falls within the strict scope of the scikit-learn API.

Indeed. There's room for a RL high-quality Python learning package.

Gaël

Sebastian Raschka

2016-03-02 19:59:17 UTC

Permalink

I am not a core developer and thus really can’t comment about the scope of scikit-learn here :P. But I am a curious about how to implement it in scikit-learn efficiently. I think an implementation based on Theano or TensorFlow may be a better place for such a module (maybe skflow, which has a scikit-like API https://github.com/tensorflow/skflow?)

Post by MichaÅ Koziarski
Hello everyone,
As far as I can tell, except PyBrain (which doesn't seem to be actively developed) there are no reinforcement learning libraries in Python. I was wondering if community would be interested in using one and making it a part of scikit-learn. Does it lie within the scope of the project?
- to design common interface, similar to what is used in other parts of scikit-sklearn;
- to implement established RL algorithms, reliant heavily on estimators available in scikit-learn;
- and to prepare practical examples of what RL can be used for, to both supplement documentation and encourage people not yet familiar with RL to experiment with it in their own projects.
Once again, I would mostly like to know whether it event lies within the scope of the project, or if it just won't be added because of project philosophy. Other than that, I would obviously appreciate any feedback.
About me: I am a last master's CS student. My research interests involve machine learning in general and reinforcement learning in particular; this year I hope to start my PhD on the latter. My master's thesis revolves around transfer learning in RL. I have experience with programming in industry and on large projects.
Cheers,
Michał
------------------------------------------------------------------------------
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140_______________________________________________
Scikit-learn-general mailing list
https://lists.sourceforge.net/lists/listinfo/scikit-learn-general

Nadim Farhat

2016-03-02 20:02:08 UTC

Permalink

I was just thinking the same but , how about just making pipelines to
Theano , TensorFlow ?

I am not a core developer and thus really canât comment about the scope of
scikit-learn here :P. But I am a curious about how to implement it in
scikit-learn efficiently. I think an implementation based on Theano or
TensorFlow may be a better place for such a module (maybe skflow, which has
a scikit-like API https://github.com/tensorflow/skflow?)

Post by MichaÅ Koziarski
Hello everyone,
As far as I can tell, except PyBrain (which doesn't seem to be actively

developed) there are no reinforcement learning libraries in Python. I was
wondering if community would be interested in using one and making it a
part of scikit-learn. Does it lie within the scope of the project?

Post by MichaÅ Koziarski
- to design common interface, similar to what is used in other parts of

scikit-sklearn;

Post by MichaÅ Koziarski
- to implement established RL algorithms, reliant heavily on estimators

available in scikit-learn;

Post by MichaÅ Koziarski
- and to prepare practical examples of what RL can be used for, to both

supplement documentation and encourage people not yet familiar with RL to
experiment with it in their own projects.

Post by MichaÅ Koziarski
Once again, I would mostly like to know whether it event lies within the

scope of the project, or if it just won't be added because of project
philosophy. Other than that, I would obviously appreciate any feedback.

Post by MichaÅ Koziarski
About me: I am a last master's CS student. My research interests involve

machine learning in general and reinforcement learning in particular; this
year I hope to start my PhD on the latter. My master's thesis revolves
around transfer learning in RL. I have experience with programming in
industry and on large projects.

Post by MichaÅ Koziarski
Cheers,
MichaÅ

------------------------------------------------------------------------------

Post by MichaÅ Koziarski
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!

http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140_______________________________________________

Post by MichaÅ Koziarski
Scikit-learn-general mailing list
https://lists.sourceforge.net/lists/listinfo/scikit-learn-general

Sebastian Raschka

2016-03-02 20:29:18 UTC

Permalink

You mean a scikit-like interface to Theano/Tensorflow? That’s actually what skflow intends to do.

I was just thinking the same but , how about just making pipelines to Theano , TensorFlow ?
I am not a core developer and thus really can’t comment about the scope of scikit-learn here :P. But I am a curious about how to implement it in scikit-learn efficiently. I think an implementation based on Theano or TensorFlow may be a better place for such a module (maybe skflow, which has a scikit-like API https://github.com/tensorflow/skflow?)

Post by MichaÅ Koziarski
Hello everyone,
As far as I can tell, except PyBrain (which doesn't seem to be actively developed) there are no reinforcement learning libraries in Python. I was wondering if community would be interested in using one and making it a part of scikit-learn. Does it lie within the scope of the project?
- to design common interface, similar to what is used in other parts of scikit-sklearn;
- to implement established RL algorithms, reliant heavily on estimators available in scikit-learn;
- and to prepare practical examples of what RL can be used for, to both supplement documentation and encourage people not yet familiar with RL to experiment with it in their own projects.
Once again, I would mostly like to know whether it event lies within the scope of the project, or if it just won't be added because of project philosophy. Other than that, I would obviously appreciate any feedback.
About me: I am a last master's CS student. My research interests involve machine learning in general and reinforcement learning in particular; this year I hope to start my PhD on the latter. My master's thesis revolves around transfer learning in RL. I have experience with programming in industry and on large projects.
Cheers,
Michał
------------------------------------------------------------------------------
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140_______________________________________________
Scikit-learn-general mailing list
https://lists.sourceforge.net/lists/listinfo/scikit-learn-general

------------------------------------------------------------------------------
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140
_______________________________________________
Scikit-learn-general mailing list
https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
------------------------------------------------------------------------------
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140_______________________________________________
Scikit-learn-general mailing list
https://lists.sourceforge.net/lists/listinfo/scikit-learn-general

Gael Varoquaux

2016-03-02 20:58:06 UTC

Permalink

Pardon me if I am saying something stupid, but isn't Theano/Tensorflow
about deep learning and not reinforcement learning. RL can be done with
deep learning, but it's more than that, and I suspect that it requires a
different API, in particular with the notion of actions.

G

Post by Sebastian Raschka
You mean a scikit-like interface to Theano/Tensorflow? That’s actually what skflow intends to do.

Post by MichaÅ Koziarski
Hello everyone,
As far as I can tell, except PyBrain (which doesn't seem to be actively developed) there are no reinforcement learning libraries in Python. I was wondering if community would be interested in using one and making it a part of scikit-learn. Does it lie within the scope of the project?
- to design common interface, similar to what is used in other parts of scikit-sklearn;
- to implement established RL algorithms, reliant heavily on estimators available in scikit-learn;
- and to prepare practical examples of what RL can be used for, to both supplement documentation and encourage people not yet familiar with RL to experiment with it in their own projects.
Once again, I would mostly like to know whether it event lies within the scope of the project, or if it just won't be added because of project philosophy. Other than that, I would obviously appreciate any feedback.
About me: I am a last master's CS student. My research interests involve machine learning in general and reinforcement learning in particular; this year I hope to start my PhD on the latter. My master's thesis revolves around transfer learning in RL. I have experience with programming in industry and on large projects.
Cheers,
Michał
------------------------------------------------------------------------------
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140_______________________________________________
Scikit-learn-general mailing list
https://lists.sourceforge.net/lists/listinfo/scikit-learn-general

--
Gael Varoquaux
Researcher, INRIA Parietal
NeuroSpin/CEA Saclay , Bat 145, 91191 Gif-sur-Yvette France
Phone: ++ 33-1-69-08-79-68
http://gael-varoquaux.info http://twitter.com/GaelVaroquaux

Kyle Kastner

2016-03-02 21:38:05 UTC

Permalink

Any RL package will have be heavily focused on non-iid data (timeseries,
basically) with the additional difficulty of the agent
effecting/interacting with the environment it is operating in. I agree with
you Gael - many packages for "deep learning" also don't handle this type of
data/these models (RNNs specifically) all that well. The models used are
also rarely "off the shelf" and generally incorporate prior knowledge about
the problem structure directly into the model, requiring customizing
connections, losses, and so on. All the things that make it really hard to
just load an object, fit, and predict.

It is a hard problem, and a good RL package would be useful. PyRL (
https://github.com/amarack/python-rl/tree/master/pyrl) and RL-Glue both
seem nice, but interacting with new environments is a classic case of "easy
to do specifically, hard to do generically". The generic case usually blows
up into defining a domain specific language (DSL), which is bad when you
are already dealing with at least one DSL (Theano/Tensorflow) if not more.

On Wed, Mar 2, 2016 at 3:58 PM, Gael Varoquaux <

Post by Gael Varoquaux
Pardon me if I am saying something stupid, but isn't Theano/Tensorflow
about deep learning and not reinforcement learning. RL can be done with
deep learning, but it's more than that, and I suspect that it requires a
different API, in particular with the notion of actions.
G

You mean a scikit-like interface to Theano/Tensorflow? Thatâs actually

what skflow intends to do.

Post by Nadim Farhat
I was just thinking the same but , how about just making pipelines to

Theano , TensorFlow ?

Post by Nadim Farhat
I am not a core developer and thus really canât comment about the

scope of scikit-learn here :P. But I am a curious about how to implement it
in scikit-learn efficiently. I think an implementation based on Theano or
TensorFlow may be a better place for such a module (maybe skflow, which has
a scikit-like API https://github.com/tensorflow/skflow?)

Post by Nadim Farhat

On Mar 2, 2016, at 2:21 PM, MichaÅ Koziarski <
Hello everyone,
As far as I can tell, except PyBrain (which doesn't seem to be

actively developed) there are no reinforcement learning libraries in
Python. I was wondering if community would be interested in using one and
making it a part of scikit-learn. Does it lie within the scope of the
project?

Post by Nadim Farhat

- to design common interface, similar to what is used in other parts

of scikit-sklearn;

Post by Nadim Farhat

- to implement established RL algorithms, reliant heavily on

estimators available in scikit-learn;

Post by Nadim Farhat

- and to prepare practical examples of what RL can be used for, to

both supplement documentation and encourage people not yet familiar with RL
to experiment with it in their own projects.

Post by Nadim Farhat

Once again, I would mostly like to know whether it event lies within

the scope of the project, or if it just won't be added because of project
philosophy. Other than that, I would obviously appreciate any feedback.

Post by Nadim Farhat

About me: I am a last master's CS student. My research interests

involve machine learning in general and reinforcement learning in
particular; this year I hope to start my PhD on the latter. My master's
thesis revolves around transfer learning in RL. I have experience with
programming in industry and on large projects.

Post by Nadim Farhat

Cheers,
MichaÅ

------------------------------------------------------------------------------

Post by Nadim Farhat

Site24x7 APM Insight: Get Deep Visibility into Application

Performance

Post by Nadim Farhat

APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!

http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140_______________________________________________

Post by Nadim Farhat

Scikit-learn-general mailing list
https://lists.sourceforge.net/lists/listinfo/scikit-learn-general

------------------------------------------------------------------------------

Post by Nadim Farhat
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140
_______________________________________________
Scikit-learn-general mailing list
https://lists.sourceforge.net/lists/listinfo/scikit-learn-general

------------------------------------------------------------------------------

http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140_______________________________________________

Post by Nadim Farhat
Scikit-learn-general mailing list
https://lists.sourceforge.net/lists/listinfo/scikit-learn-general

------------------------------------------------------------------------------

Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140
_______________________________________________
Scikit-learn-general mailing list
https://lists.sourceforge.net/lists/listinfo/scikit-learn-general

--
Gael Varoquaux
Researcher, INRIA Parietal
NeuroSpin/CEA Saclay , Bat 145, 91191 Gif-sur-Yvette France
Phone: ++ 33-1-69-08-79-68
http://gael-varoquaux.info http://twitter.com/GaelVaroquaux
------------------------------------------------------------------------------
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140
_______________________________________________
Scikit-learn-general mailing list
https://lists.sourceforge.net/lists/listinfo/scikit-learn-general

Sebastian Raschka

2016-03-02 21:54:13 UTC

Permalink

Sure, I understand.
However, I don’t see Tensorflow as a “deep learning library” only though but more of an extension of NumPy that supports symbolic expressions and efficient GPU utilization. Conveniently, it comes with tons of useful helper function that are particularly useful in the context of deep learning (e.g., all the optimizers or utility functions like tf.nn.sigmoid_cross_entropy_with_logits), but you can also use it to implement other things like SVMs, random forests; it’s up to the user.

What I am trying to say is that TensorFlow is a library (useful) *for* deep learning, not a *deep learning-only library*. So, I thought that reinforcement learning, which is typically expensive and often done via neural nets anyway, would be a better fit for a library that uses Tf or Theano vs NumPy and SciPy only. But that’s just a thought or suggestion.

Sebastian

Post by Sebastian Raschka
You mean a scikit-like interface to Theano/Tensorflow? That’s actually what skflow intends to do.

Post by MichaÅ Koziarski
Hello everyone,
As far as I can tell, except PyBrain (which doesn't seem to be actively developed) there are no reinforcement learning libraries in Python. I was wondering if community would be interested in using one and making it a part of scikit-learn. Does it lie within the scope of the project?
- to design common interface, similar to what is used in other parts of scikit-sklearn;
- to implement established RL algorithms, reliant heavily on estimators available in scikit-learn;
- and to prepare practical examples of what RL can be used for, to both supplement documentation and encourage people not yet familiar with RL to experiment with it in their own projects.
Once again, I would mostly like to know whether it event lies within the scope of the project, or if it just won't be added because of project philosophy. Other than that, I would obviously appreciate any feedback.
About me: I am a last master's CS student. My research interests involve machine learning in general and reinforcement learning in particular; this year I hope to start my PhD on the latter. My master's thesis revolves around transfer learning in RL. I have experience with programming in industry and on large projects.
Cheers,
Michał
------------------------------------------------------------------------------
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140_______________________________________________
Scikit-learn-general mailing list
https://lists.sourceforge.net/lists/listinfo/scikit-learn-general