Ph.D. Defense

Multi-Source Option-Based Policy Transfer

James MacGlashan

10:00am Friday, 25 January 2013, ITE 325B

Reinforcement learning algorithms are very effective at learning policies (mappings from states to actions) for specific well defined tasks, thereby allowing an agent to learn how to behave without extensive deliberation.  However, if an agent must complete a novel variant of a task that is similar to, but not exactly the same as, a previous version for which it has already learned a policy, learning must begin anew and there is no benefit to having previously learned anything. To address this challenge, I introduce novel approaches for policy transfer. Policy transfer allows the agent to follow the policy of a previously solved, but different, task (called a source task) while it is learning a new task (called a target task). Specifically, I introduce option-based policy transfer (OPT). OPT enables policy transfer by encapsulating the policy for a source task in an option (Sutton, Precup, & Singh 1999), which allows the agent to treat the policy of a source task as if it were a primitive action. A significant advantage of this approach is that if there are multiple source tasks, an option can be created for each of them, thereby enabling the agent to transfer knowledge from multiple sources and to combine their knowledge in useful ways. Moreover, this approach allows the agent to learn in which states of the world each source task is most applicable. OPT's approach to constructing and learning with options that represent source tasks allows OPT to greatly outperform existing policy transfer approaches. Additionally, OPT can utilize source tasks that other forms of transfer learning for reinforcement learning cannot.

Challenges for policy transfer include identifying sets of source tasks that would be useful for a target task and providing mappings between the state and action spaces of source and target tasks. That is, it may not be useful to transfer from all previously solved source tasks. If a source task has a different state or action space than the target task, then a mapping between these spaces must be provided. To address these challenges, I introduce object-oriented OPT (OO-OPT), which leverages object-oriented MDP (OO-MDP) (Diuk, Cohen, & Littman 2008) state representations to automatically detect related tasks and redundant source tasks, and to provide multiple useful state and action space mappings between tasks. I also introduce methods to adapt value function approximation techniques (which are useful when the state space of a task is very large or continuous) to the unique state representation of OO-MDPs.

Committee: Dr. Marie desJardins (Chair), Dr. Tim Finin, Dr. Michael Littman, Dr. Tim Oates, Dr. Yun Peng

i) The MIT Summer Research Program (MSRP) seeks to promote the value of graduate education; to improve the research enterprise through increased diversity; and to prepare and recruit the best and brightest for graduate education at MIT.

MSRP began in 1986 as an institutional effort to address the issue of underrepresentation of African Americans, Mexican Americans, Native Americans, and Puerto Ricans in engineering and science in the United States. Today, this program seeks to identify talented sophomores, juniors, and non-graduating seniors who might benefit from spending a summer on MIT’s campus, working in a research laboratory under the guidance of experienced scientists and engineers who are MIT faculty members, postdoctoral fellows, and advanced graduate students. Students who participate in this program will be better prepared and motivated to pursue advanced degrees, thereby helping to sustain a rich talent pool in critical areas of research and innovation.

For more information and to apply (Deadline is January 25th, 2013)  http://odge.mit.edu/undergraduate/msrp/

ii) The Columbia University IGERT on Optical Techniques for Actuation, Sensing, and Imaging of Biological Systems (BioIGERT), a National Science Foundation IGERT program, will support outstanding undergraduates as Summer Research Fellows for the period from June 2 through August 3, 2013. This program provides a $4000 stipend for each successful candidate and free campus housing (worth $2700) in which you are expected to reside. Those selected will have an opportunity to participate fully in the interdisciplinary research activities of the BioIGERT faculty across the Departments of Physics, Biology, Biomedical Engineering, Chemistry, Chemical Engineering, Mechanical Engineering, and Electrical Engineering, and includes collaborative research with industry including IBM Yorktown Heights. 

This is the best summer job you'll ever have, apply now: http://www.bioigert.columbia.edu/rpu/apply.html
For details about BioIGERT, please visit: www.bioigert.columbia.edu

iii) The Moncrief Undergraduate Summer Internship Program at the UT Austin Institute for Computational Engineering and Science is organized to provide summer support for qualified undergraduate students of mathematics, science, and engineering to work within the Institute during the summer months.

Moncrief Summer Interns work with faculty and research staff from one of ICES’ sixteen research centers and groups (full listing at: https://www.ices.utexas.edu/research/centers- groups/)
Research activities generally focus on developing modeling and simulation methods to study problems in areas such as:
 energy, advanced materials, biomedical research, nanomanufacturing, and related areas that draw on applied and computational mathematics, computing, fluid mechanics, solid mechanics, mathematical physics, and biology.
Application deadline: February 1, 2013
Application instructions at: http://www.ices.utexas.edu/programs/summer-internship/
Stipend: $6,000
Dates: Jun – Aug 2013 (10 weeks)
Who should apply: Undergraduate students of mathematics, science,or engineering who are entering their junior or senior year in Fall 2013
Requirements: 3. 0 GPA; U.S. citizens or permanent residents only


iv) The Nebraska Summer Research Program is a consortium of NSF funded Research Experiences for Undergraduates (REU) and other summer research opportunities programs (SROP).

One unique feature of the Nebraska Summer Research Program is the partnership between the REU and SROP programs and the Office of Graduate Studies. Students benefit in having a community of scholars in which to learn and to share research with.

This intensive summer research experience provides mentoring and research experiences while allowing scholars to preview graduate school life. Students with a strong interest in graduate programs are particularly encouraged to apply, as are those from populations traditionally underrepresented in graduate education.

For information on individual programs go to: http://www.unl.edu/summerprogram/

Programs: Bioenergy Systems, Biomedical Engineering, Chemistry, Earth & Atmospheric Sciences, Applied Mathematics, Nanomaterials & Nanoscience,Optics and Laser Physics, Redox Biology,Virology

Application Dates
We plan to begin accepting applications on November 14, 2012.
The priority deadline is February 1, 2013.
Complete applications are due on February 15, 2013.

Program Dates
10-week program: June 3 - August 7
8-week program: June 3 - July 26

Travel Dates
Arrival (all): Sunday, June 2
8-week program departure: Saturday, July 27
10-week program departure: Thursday, August 8

The Domestic Violence Center of Howard County will be hosting a free event at Howard Community College on Feb 19th about preventing dating violence.

Topics will include:

• Effects on health and academic achievement
• Dating and campus life
• Men and Boys taking action to end violence

Free refreshments will be provided... folks pre-registered will be eligible for prize drawings.

Photo: ua-net.eu

The ESNET Lab in UMBC's Computer Science and Electrical Engineering department is looking for qualified undergraduate or graduate students to join and work on an exciting underwater sensor networks project. For more information go to http://esnet.cs.umbc.edu/eduwan.html.

The candidates applying should have experience in (or at least familiar with) some of the following areas:

1. FPGA development using Verilog/VHDL
2. Data Frame synchronization for wireless (or acoustic) communication
3. Doppler resistant compensation for FSK modulated data
4. Digital Signal Processing
   a) Adaptive data equalization (LMS/NLMS)
   b) Error Correction coding on FPGA, i.e. turbo codes, LPDC codes.

To apply please e-mail your resume and a cover letter detailing experience in the above mentioned areas to: Sorry, you need javascript to view this email address.

Photo: ua-net.eu

The ESNET Lab in UMBC's Computer Science and Electrical Engineering department is looking for qualified undergraduate or graduate students to join and work on an exciting underwater sensor networks project. For more information go to http://esnet.cs.umbc.edu/eduwan.html.

The candidates applying should have experience in (or at least familiar with) some of the following areas:

1. FPGA development using Verilog/VHDL
2. Data Frame synchronization for wireless (or acoustic) communication
3. Doppler resistant compensation for FSK modulated data
4. Digital Signal Processing
   a) Adaptive data equalization (LMS/NLMS)
   b) Error Correction coding on FPGA, i.e. turbo codes, LPDC codes.

To apply please e-mail your resume and a cover letter detailing experience in the above mentioned areas to: Sorry, you need javascript to view this email address.

The Center for Women in Technology (CWIT) welcomes Cindy Greenwood as their new Assistant Director. Ms. Greenwood will spearhead the new Cyber Scholars Program which kicks off this fall.

Cindy Greenwood loved college so much that she never wanted to leave.

So she didn’t.  

When an Advertising and Public Relations internship during her senior year of college showed Ms. Greenwood that it wasn’t the career for her, she switched gears. She let her heart decide.

“I felt like I could make more of a difference by going into higher education,” she says.

Greenwood knows first-hand the difference that the college experience can make. Raised in Ishpeming, Michigan–a small town of no more than 7,000 people that is “half the size of UMBC,” she says—Ms. Greenwood thought it would always be her home. That is, until she left for Grand Valley State University.  

“A college campus is like no place else. You can do anything,” says Ms. Greenwood. “You can go from a cultural event where you’re trying food from Cambodia, to a dance party with glowsticks.” The atmosphere of possibility urged her to try new things, like studying abroad in Australia.

Afterwards, a master’s program in Higher Education Administration at the Leadership Center of Washington State University beckoned to her. After graduating, Ms. Greenwood spent eight years working for and with college students, first at Ferris State University in Michigan, and then at the University of South Florida.

In 2011, she joined UMBC as the Alumni Programming Coordinator in the Office of Institutional Advancement. Here, she started the Student Alumni Association to help connect current students with alumni. Hungry for more one-on-one time with undergraduates, Ms. Greenwood volunteered to be the advisor for the UMBC Vegetarian Student Group.

It’s the chance to work with students on a daily basis that drew her to the Assistant Director position in the Center for Women in Technology, she says. Ms. Greenwood will coordinate the new Cyber Scholars Program, which is run in partnership by CWIT and the UMBC Center for Cybersecurity. Her duties include overseeing the Cyber Scholars Living Learning Community, planning events, advising scholars, and teaching a seminar and bridge program for Cyber Scholars.

“The scholars programs are really interesting [at UMBC] because they really touch on every part of students’ lives,” says Greenwood.

An advocate of social justice and equality, Ms. Greenwood says she identifies with CWIT’s mission to bolster support for women in the male-centered fields of Engineering and Information Technology. She is a co-chair of UMBC’s Presidents Commission of Women. In the end it all comes back to her experience in college.

“I’ve had some good female mentors throughout my career,” says Ms. Greenwood, “and I definitely hope to be that to other females as well.”

*Ms. Greenwood joins CWIT as Assistant Director on January 28, 2013.